The Cylc Suite Engine
User Guide
5.1.1
GNU GPL v3.0 Software License
Copyright (C) 2008-2013 Hilary Oliver, NIWA
March 5, 2013
Contents
Environmental forecasting suites generate forecast products from a potentially large group of interdependent scientific models and associated data processing tasks. They are constrained by availability of external driving data: typically one or more tasks will wait on real time observations and/or model data from an external system, and these will drive other downstream tasks, and so on. The dependency diagram for a single forecast cycle in such a system is a Directed Acyclic Graph as shown in Figure 1 (in our terminology, a forecast cycle is comprised of all tasks with a common cycle time, which is the nominal analysis time or start time of the forecast models in the group). In real time operation processing will consist of a series of distinct forecast cycles that are each initiated, after a gap, by arrival of the new cycle’s external driving data.
From a job scheduling perspective task execution order in such a system must be carefully controlled in order to avoid dependency violations. Ideally, each task should be queued for execution at the instant its last prerequisite is satisfied; this is the best that can be done even if queued tasks are not able to execute immediately because of resource contention.
Cylc was developed for the EcoConnect Forecasting System at NIWA (National Institute of Water and Atmospheric Research, New Zealand). EcoConnect takes real time atmospheric and stream flow observations, and operational global weather forecasts from the Met Office (UK), and uses these to drive global sea state and regional data assimilating weather models, which in turn drive regional sea state, storm surge, and catchment river models, plus tide prediction, and a large number of associated data collection, quality control, preprocessing, post-processing, product generation, and archiving tasks.1 The global sea state forecast runs once daily. The regional weather forecast runs four times daily but it supplies surface winds and pressure to several downstream models that run only twice daily, and precipitation accumulations to catchment river models that run on an hourly cycle assimilating real time stream flow observations and using the most recently available regional weather forecast. EcoConnect runs on heterogeneous distributed hardware, including a massively parallel supercomputer and several Linux servers.
Most inter-task dependence exist within a single forecast cycle. Figure 1 shows the dependency diagram for a single forecast cycle of a simple example suite of three forecast models (a, b, and c) and three post processing or product generation tasks (d, e and f). A scheduler capable of handling this must manage, within a single forecast cycle, multiple parallel streams of execution that branch when one task generates output for several downstream tasks, and merge when one task takes input from several upstream tasks.
Figure 2 shows the optimal job schedule for two consecutive cycles of the example suite in real time operation, given execution times represented by the horizontal extent of the task bars. There is a time gap between cycles as the suite waits on new external driving data. Each task in the example suite happens to trigger off upstream tasks finishing, rather than off any intermediate output or event; this is merely a simplification that makes for clearer diagrams.
Now the question arises, what happens if the external driving data for upcoming cycles is available in advance, as it would be after a significant delay in operations, or when running a historical case study? While the forecast model a appears to depend only on the external data x at this stage of the discussion, in fact it would typically also depend on its own previous instance for the model background state used in initializing the new forecast. Thus, as alluded to in Figure 3, task a could in principle start as soon as its predecessor has finished. Figure 4 shows, however, that starting a whole new cycle at this point is dangerous - it results in dependency violations in half of the tasks in the example suite. In fact the situation is even worse than this - imagine that task b in the first cycle is delayed for any reason after the second cycle has been launched? Clearly we must consider handling inter-cycle dependence explicitly or else agree not to start the next cycle early, as is illustrated in Figure 5.
Forecast models typically depend on their own most recent previous forecast for background state or restart files of some kind, and different types of tasks in different forecast cycles can also be linked (in an atmospheric forecast analysis suite, for instance, the weather model may also generate background states for use by the observation processing and data-assimilation systems in the next cycle). In real time operation this inter-cycle dependence can be ignored because it is automatically satisfied when each cycle finishes before the next one begins. If, on the other hand, it is explicitly accounted for, it complicates the dependency graph by destroying the clean boundary between forecast cycles. Figure 6 illustrates the problem for our simple example suite assuming the minimal inter-cycle dependence: the forecast models (a, b, and c) each depend on their own previous instances.
For this reason, and perhaps because we tend to see forecasting suites as inherently sequential (with respect to whole forecast cycles) other metaschedulers ignore inter-cycle dependence and therefore require a series of distinct cycles at all times. While this does not affect normal real time operation it can be a serious impediment when advance availability of external driving data makes it possible, in principle, to run some tasks from upcoming cycles before the current cycle is finished - as suggested at the end of the previous section. This occurs after delays (late arrival of external data, system maintenance, etc.) and, to an even greater extent, in historical case studies, and parallel test suites that are delayed with respect to the main operation. It is a serious problem, in particular, for suites that have little downtime between forecast cycles and therefore take many cycles to catch up after a delay. Without taking account of inter-cycle dependence, the best that can be done, in general, is to reduce the gap between cycles to zero as shown in Figure 5. A limited crude overlap of the single cycle job schedule may be possible for specific task sets but the allowable overlap may change if new tasks are added, and it is still dangerous: it amounts to running different parts of a dependent system as if they were not dependent and as such it cannot be guaranteed that some unforeseen delay in one cycle, after the next cycle has begun, (e.g. due to resource contention or task failures) won’t result in dependency violations.
Figure 7 shows, in contrast to Figure 4, the optimal two cycle job schedule obtained by respecting all inter-cycle dependence. This assumes no delays due to resource contention or otherwise - i.e. every task runs as soon as it is ready to run. The scheduler running this suite must be able to adapt dynamically to external conditions that impact on multi-cycle scheduling in the presence of inter-cycle dependence or else, again, risk bringing the system down with dependency violations.
To further illustrate the potential benefits of proper inter-cycle dependency handling, Figure 8 shows an operational delay of almost one whole cycle in a suite with little downtime between cycles. Above the time axis is the optimal schedule that is possible in principle when inter-cycle dependence is taken into account, and below it is the only safe schedule possible in general when it is ignored. In the former case, even the cycle immediately after the delay is hardly affected, and subsequent cycles are all on time, whilst in the latter case it takes five full cycles to catch up to normal real time operation.
Similarly, Figure 9 shows example suite job schedules for an historical case study, or when catching up after a very long delay; i.e. when the external driving data are available many cycles in advance. Task a, which as the most upstream forecast model is likely to be a resource intensive atmosphere or ocean model, has no upstream dependence on co-temporal tasks and can therefore run continuously, regardless of how much downstream processing is yet to be completed in its own, or any previous, forecast cycle (actually, task a does depend on co-temporal task x which waits on the external driving data, but that returns immediately when the external data is available in advance, so the result stands). The other forecast models can also cycle continuously or with short gap between, and some post processing tasks, which have no previous-instance dependence, can run continuously or even overlap (e.g. e in this case). Thus, even for this very simple example suite, tasks from three or four different cycles can in principle run simultaneously at any given time. In fact, if our tasks are able to trigger off internal outputs of upstream tasks, rather than waiting on full completion, successive instances of the forecast models could overlap as well (because model restart outputs are generally completed early in the forecast) for an even more efficient job schedule.
Cylc manages a pool of proxy objects that represent real tasks in the forecasting suite. A task proxy can run the real task that it represents when its prerequisites are satisfied, and can receive reports of completed outputs from the real task as it runs. There is no global cycling mechanism to advance the suite in time; instead each individual task proxy has a private cycle time and spawns its own successor. Task proxies are self-contained - they just know their own prerequisites and outputs and are not aware of the wider suite context. Inter-cycle dependencies are not treated as special, and the task pool can be populated with tasks from many different cycle times. The cylc task pool is illustrated in Figure 10. Now, whenever any task changes state due to completion of an output, every task checks to see if its own prerequisites are now satisfied.2 Moreover, this matching of prerequisites and outputs involves the entire task pool, regardless of individual cycle times, so that inter- and intra-cycle dependence is handled with ease.
Thus without using global cycling mechanisms, and treating all inter-task dependence equally, cylc in effect gets a pool of tasks to self-organize by negotiating their own dependencies so that optimal scheduling, as described in the previous section, emerges naturally at run time.
The following packages are technically optional as you can construct and control cylc suites without dependency graphing, the cylc GUIs, and template processing but this is not recommended, and without Jinja2 you will not be able to run many of the example suites:
If installing via a Linux package manager, you may also need a couple of devel packages for the pygraphviz build:
Since cylc-5.0 any tagged version of cylc can be downloaded and installed for use, but you have to generate the user guide yourself from the LaTeX source (by running make). For this purpose, the following packages are also required:
And for HTML versions of the User Guide:
Finally, cylc makes heavy use of “ordered dictionary” data structures, and a significant speedup in parsing large suites can be had by installing the fast C-coded ordereddict module by Anthon van der Neut:
This module is currently included with cylc under $CYLC_DIR/ext, and is built by the top level cylc Makefile. If you install the resulting library appropriately cylc will automatically use it in place of a slower Python implementation of the ordered dictionary structure.
Cylc should run “out of the box” on recent Linux distributions.
For distributed suites the Pyro versions installed on all suite or task hosts must be mutually compatible. Using identical Pyro versions guarantees compatibility but may not be strictly necessary because cylc uses Pyro rather minimally.
Recent versions of Pyro require Python 2.5 or greater, due to use of the with statement introduced in 2.5.
Beware of Linux distributions that come packaged with old Pyro versions. Pyro 3.9 and earlier is not compatible with the new-style Python classes used in cylc. It has been reported that Ubuntu 10.04 (Lucid Lynx), released in September 2009, suffers from this problem. Surprisingly, so does Ubuntu 11.10 (Oneiric Ocelot), released in October 2011 - and therefore, presumably, earlier Ubuntu releases. Attempting to run a suite with Pyro 3.9 or earlier installed results in the following Python traceback:
It has been reported that cylc runs fine on OSX 10.6 SnowLeopard, but on OSX 10.7 Lion there is an issue with constructing proper FQDNs (Fully Qualified Domain Names) that requires a change to the DNS service. Here’s how to solve the problem:
Cylc has absorbed the following in modified form (no need to install these separately):
First install Pyro, graphviz, Pygraphviz, Jinja2, TeX, and ImageMagick using the package manager on your system if possible; otherwise download the packages manually and follow their native installation documentation. On a modern Linux system, this is very easy. For example, to install cylc-5.1.0 on the Fedora 18 Linux distribution:
If you do not have root access on your intended cylc host machine and cannot get a sysadmin to do this at system level, see Section 4.7 for some tips on installing everything to a local user account.
Cylc typically installs into a normal user account; just unpack the release tarball in the desired location (referred to below as $CYLC_DIR) and see the $CYLC_DIR/INSTALL file for instructions (see also Section G.
Cylc uses site and user configuration files to set some important global parameters such as the range of network ports, and the editor to use on suite definitions,
These files can be auto-generated with all settings intially commented out by running this command:
The content of these config files, in terms of legal items and default values, is defined by the ConfigObj configspec file,
The site config file should be adapted to set sensible defaults for all users when cylc is installed. Users can then override most settings in their own user config file, if necessary. Some settings can not be overridden by users (this is determined by #SITE ONLY comments in the configspec file, which are passed through to the config files).
Run the following command immediately after installation to copy the cylc example suites to a given destination directory, and registering each one for use:
Where TOPDIR is the top level directory into which the example suite definitions will be copied. To view the content of the resulting suite database, run cylc db print command or its GUI counterpart cylc db dbviewer:
Note that the dots in the cylc release version number are replaced with hyphens because ‘.’ is the registration name delimiter. Type cylc db print --help to see what the command options mean. Note also that the Unix “seconds since epoch” string is appended to the top level of the suite name hierarchy in order ensure uniqueness if the example suites are imported multiple times at the same cylc version: cylc-4-5-1-1344571037. You can re-register the example suites to get rid of this and make the names easier to type:
Here’s the same suite database listing in flat form:
The command cylc admin test-db gives suite registration database functionality a work out - it copies one of the cylc example suites, registers it under a new name and then manipulates it by recopying the suite in various ways, and so on, before finally deleting the test registrations. This should complete without error in a few seconds.
Cylc has a battery of self-diagnosing test suites for pre-release testing that you can also run after installation to check that everything works properly.
The command cylc admin test-battery copies, registers, and runs, the official cylc test suites, and reports the results. Some tests may take several minutes to complete. Pre-test requirements:
You must specify the hostname before running these tests:
If you don’t have access to a remote task host just use localhost as the remote hostname.
If these requirements are satisfied no tests should fail on an official cylc release, because of pre-release testing. If you find that some do fail, consider reporting this to cylc’s maintainer or the cylc mailing list. A possible exception to this is that some suite timeouts may be set too low if your suite host is slow or overloaded. Suite timeouts are intended to catch error conditions, such as unexpected task failures, that prevent a suite from completing and shutting down automatically.
If you do not have root access to your host machine and cannot easily get Pyro, graphviz, Pygraphviz, and Jinja2 installed at system level, here’s how to install everything under your home directory.
First, cylc is already designed to be installed into a normal user account - just unpack the release tarball into $CYLC_DIR. If you invoke cylc commands at this stage you will get a warning that Pyro is not installed.
Next, create a new sub-directory in the cylc source tree, $CYLC_DIR/external, and download the Pyro, Graphviz, and Pygraphviz source distributions to it (the URLs are given at the beginning of Section 3).
Install Pyro under $HOME/external/installed as follows:
Take note of the resulting Python site-packages directory under external/installed/, e.g.:
The exact path will depend on your local Python environment. Add the following to your login scripts:
Now you should be able to get cylc to print its release version:
If this command aborts and says that Pyro is not installed or is not available, then you have either not installed Pyro (check the output of the installation command carefully) or you have not pointed to the installed Pyro modules in your PYTHONPATH, or you have not sourced the cylc environment since updating PYTHONPATH.
Note that Pyro can also be installed with easy_install, which downloads and installs python packages in one shot.
At this point you should have access to all cylc functionality except for suite graphing and the gcylc graph view. For example
Install Graphviz under $CYLC_DIR/external/installed as follows:
This installs graphviz files into the bin, include, and lib sub-directories of your local installation directory. The graphviz lib and include locations are required when installing Pygraphviz (next).
Note that the graphviz build, reportedly, may fail on systems that do not have QT installed, hence the ./configure --with-qt=no option above.
Install Pygraphviz under $CYLC_DIR/external/installed as follows:
Now edit setup.py lines 31 and 32 to specify the graphviz lib and include directories:
Or you can just specify the absolute paths if you like, instead of using the $CYLC_DIR environment variable. Check that these are the correct library and include paths by inspecting the contents of the specified directories, and adjust them if necessary. Finally, install pygraphviz:
This may or may not, depending on your local Python setup, install the Pygraphviz modules into the same place as the Pyro modules, e.g.:
If not, add the correct Pyraphviz installation path to your PYTHONPATH.
The easiest way to check that pygraphviz has been installed properly is to start an interactive Python session (type python after sourcing the cylc environment to configure your PYTHONPATH) then type import pygraphviz at the interpreter prompt. If this results in an error message ImportError: No module named pygraphviz then either you have not installed pygraphviz properly, or you have not configured your PYTHONPATH to point to the installed pygraphviz modules, or you have not sourced the cylc environment since updating PYTHONPATH. Finally, if you have installed pygraphviz and configured your PYTHONPATH but graphviz itself has not been installed properly (or if the graphviz libraries have been deleted since you installed pygraphviz) then the initial pygraphiz import will be successful but a lower level import will fail when the pygraphviz modules cannot load the underlying graphviz libraries - in that case, reinstall graphviz.
You can download Jinja2 from the project web site and install it with:
or use the easy_install command to do it all in one step. Either way the final installed package location must be present in the PYTHONPATH variable, and you may have to arrange for this first. You may also need to create the installed package directory if it doesn’t exist already (if so the install will abort and print the name of the missing directory in the error message). Here’s how to easy_install Jinja2 into your new private python site packages directory:
Adapt the site-packages path according to your actual path, as above.
You should now have access to all cylc functionality. Import the example suites if you have not done so already (Section 4.4) then test your cylc installation by running the automated suite database test (Section 4.5) and the automated scheduler test (Section 4.6), then go on to the Quick Start Guide (Section 6).
Upgrading is just a matter of unpacking the new cylc release and optionally re-importing the example suites for the new version.
From using other schedulers you may be accustomed to the idea that a forecasting suite has a “current cycle time”, which is typically the analysis time or nominal start time of the main forecast model(s) in the suite, and that the whole suite advances to the next forecast cycle when all tasks in the current cycle have finished (or even when a particular wall clock time is reached, in real time operation). As is explained in the Introduction, this is not how cylc works.
Cylc suites advance by means of individual tasks with private cycle times independently spawning successors at the next valid cycle time for the task, not by incrementing a suite-wide forecast cycle. Each task will be submitted when its own prerequisites are satisfied, regardless of other tasks with other cycle times running, or not, at the time. It may still be convenient at times, however, to refer to the “current cycle”, the “previous cycle”, or the “next cycle” and so forth, with reference to a particular task, or in the sense of all tasks that “belong to” a particular forecast cycle. But keep in mind that the members of these groups may not be present simultaneously in the running suite - i.e. different tasks may pass through the “current cycle” (etc.) at different times as the suite evolves, particularly in delayed (catch up) operation.
This section works through some basic cylc functionality using the “QuickStart” example suites, which you can import to your suite database by running the cylc admin import-examples command and then reregistering the top level suite name to “examples” as described in Section 4.4. You should end up with the following QuickStart suites (but the directory paths on the right are up to you):
Cylc suites are defined by suite.rc files, discussed at length in Suite Definition (Section 8) and the Suite.rc Reference (Appendix A). To view the examples.QuickStart.a suite definition right-click on the suite name and choose ‘Edit’; or use the edit command:
This opens the suite definition in your editor (configured in the cylc site or user config file - see Section 4.3) from the suite definition directory so that you can easily open other suite files in the editor. You can of course do this manually, but by using the cylc interface you don’t have to remember suite directory locations. If you do need to move to a suite definition directory, you can do this:
Suites that use include-files can optionally be edited in a temporarily inlined state - the inlined file gets split back into its constituent include-files when you save it and exit the editor. While editing, the inlined file becomes the official suite definition so that changes take effect whenever you save the file.
Anyhow, you should now see the following suite.rc file in your editor:
Cylc comes with syntax highlighting and section folding for the vim editor, and an emacs font-lock mode - see Section 8.2.3.
This defines a complete, valid, runnable suite. Here’s how to interpret it: At 0, 6, 12, and 18 hours each day a clock-triggered task called GetData triggers 1 hour after the wall clock reaches its (GetData’s) nominal cycle time; then a task called Model triggers when GetData finishes; and a task called PostA triggers when Model is finished. Additionally, Model depends on its own previous instance from 6 hours earlier; and twice per day at 6 and 18 hours another task called PostB also triggers off Model.
All the tasks in this suite can run in parallel with their own previous instances if the opportunity arises (i.e. if their prerequisites are satisfied before the previous instance is finished). Most tasks should be capable of this (see Section 14.4) but if necessary you can force particular tasks to run sequentially like this:
Finally, when the suite is cold-started (started from scratch) it is made to wait on a special synchronous start-up task called Prep. Start-up tasks are one-off (non-spawning) tasks that are only used at suite start-up, and any dependence on them only applies at suite start-up. They cannot be used in conditional trigger expressions with normal cycling tasks, because the trigger becomes undefined in subsequent cycles. Start-up tasks are synchronous because they have a defined cycle time even though they are not cycling tasks. Cylc also has asynchronous one-off tasks, which have no cycle time:
The optional visualization section configures graph plotting.
Right-click on the examples.QuickStart.a suite in the db viewer and choose Graph; or by command line,
This will pop up a zoomable, pannable, graph viewer showing the graph of Figure 16. If you edit the suite.rc file the viewer will update in real time whenever you save the file.
Each cylc task defines command scripting to invoke the right external processing when the task is ready to run. This has not been explicilty configured in the example suite, so it defaults, for all tasks, to the dummy task scripting inherited from the root namespace (see Section 8):
The command arguments above reflect suite definition section nesting.
Now start a suite control GUI by right-clicking on the suite in the db viewer and choosing Control → Graph View. You can also open other control GUIs for the same suite if you like. Multiple GUIs running at the same time will automatically connect to the same running suite (they won’t try to run separate instances). Note also that if you shut down a suite control GUI, the suite will keep running. You can reconnect to it later by opening another control GUI.
In the control GUI click on Control → Run, enter an initial cold-start cycle time (e.g. 2011052306), and select “Hold (pause) on start-up” so that the suite will start in the held state (tasks will not be submitted even if they are ready to run).
Do not choose an initial cycle time in the future unless you’re running in simulation mode, or nothing much will happen until that time.
If the initial cycle time ends in 06 or 18 the suite controller should look like Figure 17, or otherwise (00 or 12) like Figure 18.
The reason for the difference in graph structure between the two figures is this: cylc starts up with every task present in the waiting state (blue) at the initial cycle time or at the first subsequent valid cycle time for the task - and PostB does not run at 00 or 12. The greyed out tasks are from the base graph, defined in the suite.rc file, and aren’t actually present in the suite as yet (they are shown in the graph in order to put the live tasks in context).
Now, click on Control → Release in the suite control GUI to release the hold on the suite, and observe what happens: the GetData tasks will rapidly go off in parallel out to a few cycles ahead (how far ahead depends on the suite runahead limit as explained below and in The Suite Runahead Limit, Section 11.6.1) and then the suite will stall, as shown in Figures 19 and 20.
The Prep task runs immediately because it has no prerequisites and is not clock-triggered. The clock-triggered GetData tasks then all go off at once because they have no prerequisites (i.e. they do not have to wait on any upstream tasks), their trigger time has long passed (the initial cycle time was in the past), and they are not sequential tasks (so they are able to run in parallel - try declaring GetData sequential to see the difference). Beyond the suite runahead limit which is set to 12 hours in this suite - see Section 11.6.1) GetData is put into a special ‘runahead’ held state indicated by the darker blue graph node. The task will be released from this state when the slower tasks have caught up sufficiently. The runahead limit is designed to stop quick tasks from running off too far into the future. It is typically of little consequence in real time operation when suites are typically constrained by clock triggered tasks.
If you’re wondering why a particular task has not triggered yet in a running suite you can view the current state of its prerequisites by right-clicking on the task and choosing ‘View State’, or using cylc show. Do this for the first Model task, which appears to be stuck in the waiting state; it will pop up a small window as in Figure 21.
It is clear that the reason the task is not running, and consequently, by virtue of the runahead limit, why the suite has stalled, is that Model[T] is waiting on Model[T-6] which does not exist at suite start-up. Model represents a warm-cycled forecast model that depends on a model background state or restart file(s) generated by its own previous run.
Right-click on the waiting Model task and choose Trigger, or use cylc trigger, to force the task to trigger and thereby get the suite up and running. In a real suite this would not be sufficient: the real forecast model that Model represents would fail for lack of the real restart files that it requires as input. We’ll see how to handle this properly shortly.
After watching the examples.QuickStart.a suite run for a while choose Stop from the Control menu, or cylc stop, to shut it down. The default stop method waits for any tasks that are currently running to finish before shutting the suite down, so that the final recorded suite state is perfectly consistent with what actually happened.
You can restart the suite from where it left off by choosing Control → Run and selecting the ‘restart’ option, or using cylc restart. Note that cylc always writes a special state dump, and logs its name, prior to actioning any intervention, and you can also restart a suite from one of these states, rather than the default most recent state.
Now take a look at examples.QuickStart.b, which is a minor modification of examples.QuickStart.a. Its suite.rc file has a new cold-start task called ColdModel,
and the dependency graph (see also Figure 22) looks like this:
In other words, Model[T] can trigger off either Model[T-6] or ColdModel[T].
Cold-start tasks are one-off tasks used in the first cycle to satisfy another task’s intercycle-cycle dependence at suite start-up (when there is no previous cycle to do it). For instance, a series of cold-start tasks may be used to cold-start a warm-cycled model. Unlike start-up tasks though, cold-start dependence is preserved in subsequent cycles, so they must generally appear in OR’d conditional triggers in order to avoid stalling the suite after the first cycle (as in this example). This means cold-start tasks can be inserted into a running suite, if necessary, to cold-start their associated tasks in case of problems that prevent continued normal warm cycling.
A cold-start task in a real suite may submit a real “cold start forecast”, or similar, to generate the previous-cycle input files required by the associated model, or it may just stand in for some external spinup process, or similar, that has to be completed before the suite is started (in the latter case the cold-start task would be a dummy task that just reports successful completion in order to satisfy the initial previous-cycle dependence of the model).
Run examples.QuickStart.b to confirm that that no manual triggering is required to get the suite started now.
The suite examples.QuickStart.c is the same as examples.QuickStart.b except that it has real task implementations (scripts located in the suite bin directory) that generate and consume files in such a way that they have to run according to the graph of Figure 22. The suite gets them to run together out of a common I/O workspace, configured via the suite.rc file.
By studying this suite and its tasks, and by making quick copies of it to modify and run, you should be able to learn a lot about how to build real cylc suites. Here’s the complete suite definition
Here’s the namespace hierarchy defined by this suite:
And here, for example, is the complete implementation for the PostA task (located with the other task scripts in the suite bin directory):
Cylc writes some information, including warnings and errors, to the suite stdout stream. In debug mode (cylc run --debug) this output is directed to the terminal; otherwise it is directed to a log file under the suite run directory.
The suite event log records timestamped events as the suite runs; it is stored under the suite run directory.
Figure 23 shows a suite log viewed from gcylc. The cylc log command also prints the suite event, stdout, and stderr logs, with optionally filtering of the event log for specific tasks.
The stdout and stderr streams from running tasks (if they do not detach from the process that does the initial job submission, or manage their own output) are written to a suite-specific sub-directory of the suite run directory. The location of this directory is determined by site/user config files, defaulting to $HOME/cylc-run/$CYLC_SUITE_REG_NAME/log/job/ (where $CYLC_SUITE_REG_NAME is the registered suite name).
The cylc suite search tool reports matches in the suite.rc file by line number, suite section, and file, even if include-files are used (and even if they are nested), and by file and line number for matches in the suite bin directory. The following output listing is from a search of the examples.QuickStart.c suite.
(Suite search is also available from the db viewer right-click menu).
The cylc diff command compares suites and reports differences by suite.rc section and item. Note that some differences may be due to suite-name-specific defaults that are not explicitly configured in either suite.
Suite validation checks for errors by parsing the suite definition, comparing all items against the suite.rc specification file, and then parsing the suite graph and attempting to instantiate all task proxy objects. This can be done using the cylc GUIs or cylc validate:
For more information on suite validation see Section 8.2.5.
Cylc has been designed from the ground up to make prototyping and testing new suites very easy. Simply drawing (in text) a dependency graph in a new suite definition creates a valid suite that you can run: the tasks will be dummy tasks that default to emitting an identifying message, sleeping for a few seconds, and then exiting; but you can then arrange for particular tasks to do more complex things by configuring their runtime properties appropriately.
Cylc has example suites intended to illustrate most facets of suite construction. These are held centrally under $CYLC_DIR/examples and can be imported to your suite database by running ’cylc admin import-examples’. They all run “out the box” and can be copied and modified by users to test almost anything. Some of them just configure a suite dependency graph, in which case cylc will run dummy tasks according to the graph; some also configure task runtime properties (e.g. command scripting and environment variables) within the suite definition; and some have real task implementations that generate and consume real files and which will fail if they are not executed in the right order. All of the example suites are portable in the sense that all suite and task I/O directory paths incorporate the suite registration name (this is the default situation for any cylc suite in fact) so you can run multiple copies of the same suite at once without any interference between them.
(This suite is explained in the Quick Start Guide, Section 6).
When running a suite in live mode that contains clock-triggered tasks do not give an initial cycle time in the future or nothing will happen until that time. However, you can also run any suite in simulation mode or dummy mode in which case a future start time is fine (see Appendix 11.13).
Cylc commands target particular suites via names registered in a suite database, so that you don’t need to remember and continually refer to the actual location of the suite definition on disk. A suite registration name is a hierarchical name akin to a directory path but delimited by the ‘.’ character; this allows suites to be organised in nested tree-like structures:
Note that registration groups are entirely virtual, they do not need to be explicitly created before use, and they automatically disappear if all tasks are removed from them. From the listing above, for example, to move the suite nwp.oper.region2 into the nwp.test group:
And to move nwp.test.region2 into a new group nwp.para:
Currently you cannot explicitly indicate a group name on the command line by appending a dot character. Rather, in database operations such as copy, reregister, or unregister, the identity of the source item (group or suite) is inferred from the content of the database; and if the source item is a group, so must the target be a group (or it will be, in the case of an item that will be created by the operation). This means that you cannot copy a single suite into a group that does not exist yet unless you specify the entire target registration name.
cylc db register --help shows a number of other examples.
Any client process that connects to a running suite (this includes task messaging and user-invoked interrogation and control commands) must authenticate with a secure passphrase that has been loaded by the suite. A random passphrase is generated automatically in the suite definition directory at registration time, if one does not already exist there. For the default Pyro-based connection method the passphrase file must be distributed to any other accounts that host running tasks or from which you need monitoring or control access to the running suite. Alternatively, an ssh-based communication method can be used to automatically re-invoke cylc commands, including task messaging, on the suite host, in which case the suite passphrase is only needed on the suite host. See Section 11.1.1 for more on how cylc’s client/server communication works and how to use it.
Each user has a suite database that associates registered suite names with their respective suite definition directory locations. Hierarchical suite names stored in the database can be displayed in a tree structure. By right-clicking on a suite in your database, from within the db viewer, or using cylc commands, you can:
Note that the suite title shown in the db viewer is parsed from the suite.rc file at the time of initial registration; if you change the title (by editing the suite.rc file) use cylc db refresh or the db viewer’s View → Refresh to update the database.
The user suite database file is $HOME/.cylc/DB.
On the command line, the ‘database’ (or ‘db’) command category contains commands to implement the aforementioned operations.
Groups of suites (at any level in the registration hierarchy) can be deleted, copied, imported, and exported; as well as individual suites. To do this, just use suite group names as source and/or target for operations, as appropriate. For instance, if a group foo.bar contains the suites foo.bar.baz and foo.bar.qux, you can copy a single suite like this:
(resulting in a new suite boo); or the group like this:
(resulting in new suites boo.baz and boo.qux); or the group like this:
(resulting in new suites boo.bar.baz and boo.bar.qux). When suites are copied, the suite definition directories are copied into a directory tree, under the target directory, that reflects the registration name hierarchy. cylc copy --help has some explicit examples.
The same functionality is also available by right-clicking on suites or suite groups in the db viewer GUI, as shown in Figure 11.
Cylc suites are defined in structured, validated, suite.rc files that concisely specify the properties of, and the relationships between, the various tasks managed by the suite. This section of the User Guide deals with the format and content of the suite.rc file, including task definition. Task implementation - what’s required of the real commands, scripts, or programs that do the processing that the tasks represent - is covered in Section 9; and task job submission - how tasks are submitted to run - is in Section 10.
A cylc suite definition directory contains:
A typical example:
Suite.rc files conform to the ConfigObj extended INI format (http://www.voidspace.org.uk/python/configobj.html) with several modifications to allow continuation lines and include-files, and to make it legal to redefine environment variables and scheduler directives (duplicate config item definitions are normally flagged as an error).
Additionally, embedded template processor expressions may be used in the file, to programatically generate the final suite definition seen by cylc. Currently the Jinja2 template engine is supported (http://jinja.pocoo.org/docs). In the future cylc may provide a plug-in interface to allow use of other template engines too. See Jinja2 Suite Templates (Section 8.6) for some examples.
The following list shows legal raw suite.rc syntax. Suites using the Jinja2 template processor (Section 8.6) can of course use Jinja2 syntax as well (it must generate raw syntax on processing).
The following pseudo-listing illustrates suite.rc syntax:
Cylc has native support for suite.rc include-files, which may help to organize large suites. Inclusion boundaries are completely arbitrary - you can think of include-files as chunks of the suite.rc file simply cut-and-pasted into another file. Include-files may be included multiple times in the same file, and even nested. Include-file paths can be specified portably relative to the suite definition directory, e.g.:
Editing Temporarily Inlined Suites Cylc’s native file inclusion mechanism supports optional inlined editing:
The suite will be split back into its constituent include-files when you exit the edit session. While editing, the inlined file becomes the official suite definition so that changes take effect whenever you save the file. See cylc prep edit --help for more information.
Include-Files via Jinja2 Jinja2 (Section 8.6) provides template inclusion functionality although this is more akin to Python module import than simple text inclusion, and the implications of this for suite design have not yet been explored.
Cylc comes with a syntax file to configure suite.rc syntax highlighting and section folding in the vim editor, as shown in Figure 12, and an emacs font-lock mode. Both are stored under the cylc conf directory:
Refer to comments at the top of each file to see how to use them.
Cylc suite.rc files consist of a suite title and description followed by configuration items grouped under several top level section headings:
Cylc suite.rc files are automatically validated against a specification that defines all legal entries, values, options, and defaults (held in $CYLC_DIR/conf/suiterc/). This detects any formatting errors, typographic errors, illegal items and illegal values prior to run time. Some values are complex strings that require further parsing by cylc to determine their correctness (this is also done during validation). All legal entries are documented in the Suite.rc Reference (Appendix A).
The validator reports the line numbers of detected errors. Here’s an example showing a subsection heading with a missing right bracket.
If the suite.rc file contains include-files you can use cylc view to view an inlined copy with correct line numbers (you can also edit suites in a temporarily inlined state with cylc edit --inline).
Validation does not check the validity of chosen job submission methods; this is to allow users to extend cylc with their own job submission methods, which are by definition unknown to the suite.rc spec.
The [scheduling] section of a suite.rc file defines the relationships between tasks in a suite - the information that allows cylc to determine when tasks are ready to run. The most important component of this is the suite dependency graph. Cylc graph notation makes clear textual graph representations that are very concise because sections of the graph that repeat at different hours of the day, say, only have to be defined once. Here’s an example with dependencies that vary depending on cycle time:
Figure 24 shows the complete suite.rc listing alongside the suite graph. This is actually a complete, valid, runnable suite (it will use default runtime properties and command scripting and you’ll need to trigger task A manually to get the suite started because A[T] depends on A[T-6] and at start-up there is no previous cycle to satisfy that dependence - how to handle this properly is described in Handling Intercycle Dependencies At Start-Up (Section 8.3.5) and in the Quick Start Guide (Section 6).
Multiline graph strings may contain:
Suite dependency graphs can be broken down into pairs in which the left side (which may be a single task or family, or several that are conditionally related) defines a trigger for the task or family on the right. For instance the “word graph” C triggers off B which triggers off A can be deconstructed into pairs C triggers off B and B triggers off A. In this section we use only the default trigger type, which is to trigger off the upstream task succeeding; see Section 8.3.4 for other available triggers.
In the case of cycling tasks, the triggers defined by a graph string are valid for cycle times matching the list of hours specified for the graph section. For example this graph,
implies that B triggers off A for cycle times in which the hour matches 0 or 12.
To define intercycle dependencies, attach an offset indicator to the left side of a pair:
This means B[T] triggers off A[T-12] for cycle times T with hours matching 0 or 12. Note that T must be left implicit unless there is a cycle time offset (this helps to keep graphs clean and concise because the majority of tasks in a typical suite will only depend on others with the same cycle time) and that cycle time offsets can only appear on the left (because each pair defines a trigger for the right task at cycle time T).
Now, having explained that dependency graphs are interpreted pairwise, you can optionally chain pairs together to “follow a path” through the graph. So this,
is equivalent to this:
Cycle time offsets, if they appear in a chain of triggers, must be leftmost (because, as explained previously they can’t appear on the right of any pair). So this is legal:
but this isn’t:
The trigger A => B[T-6] does not make sense in any case - if this kind of relationship seems necessary it probably means that B should be “reassigned” to the next cycle (keep in mind that cycle time is really just a label used to define the relatonships between tasks).
Each trigger in the graph must be unique but the same task can appear in multiple pairs or chains. Separately defined triggers for the same task have an AND relationship. So this:
is equivalent to this:
In summary, the branching tree structure of a dependency graph can be partitioned into lines (in the suite.rc graph string) of pairs or chains, in any way you like, with liberal use of internal white space and comments to make the graph structure as clear as possible.
Handling Long Graph Lines Long chains of dependencies can be split into pairs:
If you have very long task names, or long conditional trigger expressions (below) then you can use the suite.rc line continuation marker:
Note that a line continuation marker must be the final character on the line; it cannot be followed by trailing spaces or a comment.
A suite definition can contain multiple graph strings that are combined to generate the final graph. There are different graph VALIDITY section headings for cycling, one-off asynchronous, and repeating asynchronous tasks. Additionally, there may be multiple graph strings under different VALIDITY sections for cycling tasks with different dependencies at different cycle times.
One-off Asynchronous Tasks Figure 25 shows a small suite of one-off asynchronous tasks; these have no associated cycle time and don’t spawn successors (once they’re all finished the suite just exits). The integer 1 attached to each graph node is just an arbitrary label, akin to the task cycle time in cycling tasks; it increments when a repeating asynchronous task (below) spawns.
Cycling Tasks For cycling tasks the graph VALIDITY section heading defines a sequence of cycles times for which the subsequent graph section is valid. Figure 26 shows a small suite of cycling tasks.
Stepped Daily, Monthly, And Yearly Cycling In addition to the original hours-of-the-day section headings, cylc now has an extensible cycling mechanism and (so far) stepped daily, monthly, and yearly cycling modules:
In the examples above the section headings define an infinite sequence of cycle times anchored on the first (date-time) argument and stepped by the second (integer) argument. The anchoring serves to generate the same sequence, as opposed to some off-set sequence, regardless of the initial cycle time from which the suite is started. The anchor date can lie outside of the suite’s initial and final cycle times.
Note that hours-of-the-day graph section headings can also be written to explicitly reference the associated cycling module:
How Multiple Graph Strings Combine For a cycling graph with multiple validity sections for different hours of the day, the different sections add to generate the complete graph. Different graph sections can overlap (i.e. the same hours may appear in multiple section headings) and the same tasks may appear in multiple sections, but individual dependencies should be unique across the entire graph. For example, the following graph defines a duplicate prerequisite for task C:
This does not affect scheduling, but for the sake of clarity and brevity the graph should be written like this:
Combined Asynchronous And Synchronous Graphs Cycling tasks can be made to wait on one-off asynchronous tasks, as shown in Figure 27. Alternatively, they can be made to wait on one-off synchronous start-up tasks, which have an associated cycle time even though they are non-cycling - see Figure 28.
Synchronous Start-up vs One-off Asynchronous Tasks One-off synchronous start-up tasks run only when a cycling suite is cold-started and they are often associated with subsequent one-off cold-start tasks used to bootstrap a cycling suite into existence.
The distinction between cold- and warm-start is only meaningful for cycling tasks, and one-off asynchronous tasks may be best used in constructing entirely non-cycling suites.
However, one-off asynchronous tasks can precede cycling tasks in the same suite, as shown above. It seems likely that, if used in this way, they will be intended as start-up tasks - so currently one-off asynchronous tasks only run in a cold-start.
Repeating Asynchronous Tasks Repeating asynchronous tasks can be used, for example, to process satellite data that arrives at irregular time intervals. Each new dataset must have a unique “asynchronous ID”. If it doesn’t naturally have such an ID a string representation of the data arrival time could be used. The graph VALIDITY section heading must contain “ASYNCID:” followed by a regular expression that matches the actual IDs. Additionally, one task in the suite must be a designated “daemon” that waits indefinitely on incoming data and reports each new dataset (and its ID) back to the suite by means of a special output message. When the daemon task proxy receives a matching message it dynamically registers a new output (containing the ID) that downstream tasks can then trigger off. The downstream tasks likewise have prerequisites containing the ID pattern (because they trigger off the aforementioned outputs) and when these get satisfied during dependency negotiation the actual ID is substituted into their own registered outputs. Finally, each asynchronous repeating task proxy passes the ID to its task execution environment as $ASYNCID to allow identification of the correct dataset by task scripts. In this way a tree of tasks becomes dedicated to processing each new dataset, and multiple datasets can be processed in parallel if they become available in quick succession. As Figure 29 shows, a repeating asynchronous suite currently plots just like a one-off asynchronous suite. But at run time the daemon task stays put, while the others continually spawn successors to wait for new datasets to come in. The asynchronous.repeating example suite demonstrates how to do this in a real suite. Note that other trigger types (success, failure, start, suicide, and conditional) cannot currently be used in a repeating asynchronous graph section.
Trigger type, indicated by :type after the upstream task (or family) name, determines what kind of event results in the downstream task (or family) triggering.
Success Triggers The default, with no trigger type specified, is to trigger off the upstream task succeeding:
For consistency and completeness, however, the success trigger can be explicit:
Failure Triggers To trigger off the upstream task reporting failure:
Section 8.3.4.8 (Suicide Triggers) shows one way of handling task B here if A does not fail.
Start Triggers To trigger off the upstream task starting to execute:
This can be used to trigger tasks that monitor other tasks once they (the target tasks) start executing. Consider a long-running forecast model, for instance, that generates a sequence of output files as it runs. A postprocessing task could be launched with a start trigger on the model (model:start => post) to process the model output as it becomes available. Note, however, that there are several alternative ways of handling this scenario: both tasks could be triggered at the same time (foo => model & post), but depending on external queue delays this could result in the monitoring task starting to execute first; or a different postprocessing task could be triggered off an internal output for each data file (model:out1 => post1 etc.; see Section 8.3.4.5), but this may not be practical if the number of output files is large or if it is difficult to add cylc messaging calls to the model.
Finish Triggers To trigger off the upstream task succeeding or failing, i.e. finishing one way or the other:
Internal Triggers These are only required to trigger off events that occur before a task finishes.
Task A must emit this message when the actual output has been completed - see Reporting Internal Outputs Completed (Section 9.4.2).
Intercycle Triggers Typically most tasks in a suite will trigger off other cotemporal (i.e. the same cycle time) tasks, but some may depend on tasks with earlier cycle times. This notably applies to warm-cycled forecast models, which depend on their own previous instances (see below); but other kinds of intercycle dependence are possible too.5 Here’s how to express this kind of relationship in cylc:
Intercycle and trigger type (and internal output) notation can be combined:
Bootstrapping Intercycle Triggers Tasks with intercycle triggers require an associated cold-start task to bootstrap them into operation when the suite is cold-started, because they depend on a previous cycle that does not exist at start time. Otherwise the first such task will require manual triggering (and that will only suffice if the real task does not have real previous-cycle dependence in the first cycle). Section 8.3.5, Handling Intercycle Dependence At Start-Up, explains how to use cold-start tasks in cylc.
Conditional Triggers AND operators (&) can appear on both sides of an arrow. They provide a concise alternative to defining multiple triggers separately:
OR operators (|) which result in true conditional triggers, can only appear on the left,6
Forecasting suites typically have simple conditional triggering requirements, but any valid conditional expression can be used, as shown in Figure 30 (conditional triggers are plotted with open arrow heads).
Suicide Triggers Suicide triggers take tasks out of the suite. This can be used for automated failure recovery. The suite.rc listing and accompanying graph in Figure 31 show how to define a chain of failure recovery tasks that trigger if they’re needed but otherwise remove themselves from the suite (you can run the AutoRecover.async example suite to see how this works). The dashed graph edges ending in solid dots indicate suicide triggers, and the open arrowheads indicate conditional triggers as usual.
Note that multiple suicide triggers combine in the same way as other triggers, so this:
is equivalent to this:
i.e. both foo and bar must succeed for baz to be taken out of the suite. If you really want a task to be taken out if any one of several events occurs then be careful to write it that way:
Family Triggers Families defined by the namespace inheritance hierarchy (Section 8.4) can be used in the graph trigger whole groups of tasks at the same time (e.g. forecast model ensembles and groups of tasks for processing different observation types at the same time) and for triggering downstream tasks off families as a whole. Higher level families, i.e. families of families, can also be used, and are reduced to the lowest level member tasks. Note that tasks can also trigger off individual family members if necessary.
To trigger an entire task family at once:
This is equivalent to:
To trigger other tasks off families we have to specify whether to triggering off all members starting, succeeding, failing, or finishing, or off any members (doing the same). Legal family triggers are thus:
Here’s how to trigger downstream processing after if one or more family members succeed, but only after all members have finished (succeeded or failed):
In suites with intercycle dependence some kind of bootsrapping process is required to get the suite going initially. In the example shown in Intercycle Triggers (Section 8.3.4.6), for instance, in the very first cycle there is no previous instance of task A to satisfy B’s prerequisites.
Cold-Start Tasks A cold-start task is a special one-off task used to satisfy the initial previous-cycle dependence of another cotemporal task. In effect, the cold-start task masquerades as the previous-cycle trigger of its associated cycling task.
A cold-start task may invoke real processing to generate the files that would normally be produced by the associated cycling task; or it could be a dummy task that represents some external spinup process that presumably generates in the same files but which has to be completed before the suite is started. In the latter case the cold-start task can just report itself successfully completed after checking that the required files are present.
This kind of relationship can easily be expressed with a conditional trigger:
i.e. Foo[T] can trigger off either Bar[T-6] or ColdFoo[T]. At start-up ColdFoo will do the job, and thereafter Bar[T-6] will do it.
Cold-start tasks can also be inserted into the suite at run time to cold-start just their associated cycling tasks, if a problem of some kind prevents continued normal cycling.
Warm-Starting A Suite Cold-start tasks have to be declared as such in the suite.rc “special tasks” section so that cylc knows they are one-off (non-spawning) tasks, but also because they play a critical role in suite warm-starts. A suite that has previously been running and then shut down can be warm-started at a particular cycle time, an alternative to restarting from a previous state (although restarting is preferred because a warm start is likely to involve re-running some tasks). A warm-start assumes the existence of a previous cycle (i.e. that any files from the previous cycle required by the new cycle are in place already) so cold-start tasks do not need to run but cylc itself does not know the details of the previous cycle (it does in a restart, but not in a warm-start) so it still has to solve the bootstrapping problem to get the suite started. It does this by starting the suite with designated cold start tasks in the succeeded state - in other words finished cold start tasks stand in for the previous finished cycle, rather than pretending to be a running previous cycle as they do in a cold-start.
Warm cycled forecast models generate restart files, e.g. model background fields, that are required to initialize the next forecast (this is essentially the definition of “warm cycling”). In fact restart files will often be written for a whole series of subsequent cycles in case the next cycle (or the next and the next-next, and so on) cycle has to be omitted:
In other words, task A can trigger off a cotemporal cold-start task, or off its own previous instance, or off the instance before that, and so on. Restart dependencies are unusual because although A could trigger off A[T-12] we don’t actually want it to do so unless A[T-6] fails and can’t be fixed. This is why Task A, above, is declared to be ‘sequential’.7 Sequential tasks do not spawn a successor until they have succeeded (by default, tasks spawn as soon as they start running in order to get maximum functional parallelism in a suite) which means that A[T+6] will not be waiting around to trigger off an older predecessor while A[T] is still running. If A[T] fails though, the operator can force it, on removal, to spawn A[T+6], whose restart dependencies will then automatically be satisfied by the older instance, A[T-6].
Forcing a model to run sequentially means, of course, that its restart dependencies cannot be violated anyway, so we might just ignore them. This is certainly an option, but it should be noted that there are some benefits to having your suite reflect all of the real dependencies between the tasks that it is managing, particularly for complex multi-model operational suites in which the suite operator might not be an expert on the models. Consider such a suite in which a failure in a driving model (e.g. weather) precludes running one or more cycles of the downstream models (sea state, storm surge, river flow, …). If the real restart dependencies of each model are known to the suite, the operator can just do a recursive purge to remove the subtree of all tasks that can never run due to the failure, and then cold-start the failed driving model after a gap (skipping as few cycles as possible until the new cold-start input data are available). After that the downstream models will kick off automatically so long as the gap is spanned by their respective restart files, because their restart dependencies will automatically be satisfied by the older pre-gap instances in the suite. Managing this kind of scenario manually in a complex suite can be quite difficult.
Finally, if a warm cycled model is declared to have explicit restart outputs, and is not declared to be sequential, and you define appropriate labeled restart outputs which must contain the word ‘restart’, then the task will spawn as soon its last restart output is completed so that successives instances of the task will be able to overlap (i.e. run in parallel) if the opportunity arises. Whether or not this is worth the effort depends on your needs.
The [runtime] section of a suite definition configures what to execute (and where and how to execute it) when each task is ready to run, in a multiple inheritance hierarchy of namespaces culminating in individual tasks. This allows all common configuration detail to be factored out and defined in one place.
Any namespace can configure any or all of the items defined in the Suite.rc Reference, Appendix A.
Namespaces that do not explicitly inherit from others automatically inherit from the root namespace (below).
Nested namespaces define task families that can be used in the graph as convenient shorthand for triggering all member tasks at once, or for triggering other tasks off all members at once - see Family Triggers, Section 8.3.4.9. Nested namespaces can be progressively expanded and collapsed in the dependency graph viewer, and in the gcylc graph and tree views. Only the first parent of each namespace (as for single-inheritance) is used for suite visualization purposes.
Namespace names may contain letters, digits, underscores, and hyphens.
Note that task names need not be hardwired into task implementations because task and suite identity can be extracted portably from the task execution environment supplied by cylc (Section 8.4.7) - then to rename a task you can just change its name in the suite definition.
The root namespace, at the base of the inheritance hierarchy, provides default configuration for all tasks in the suite. Most root items are unset by default, but some have default values sufficient to allow test suites to be defined by dependency graph alone. The command scripting item, for example, defaults to code that prints a message then sleeps for between 1 and 15 seconds and exits. Default values are documented with each item in Appendix A. You can override the defaults or provide your own defaults by explicitly configuring the root namespace.
If a namespace section heading is a comma-separated list of names then the subsequent configuration applies to each list member. Particular tasks can be singled out at run time using the $CYLC_TASK_NAME variable.
As an example, consider a suite containing an ensemble of closely related tasks that each invokes the same script but with a unique argument that identifies the calling task name:
For large ensembles Jinja2 template processing can be used to automatically generate the member names and associated dependencies (see Section 8.6).
The following listing of the inherit.single.one example suite illustrates basic runtime inheritance with single parents.
If a namespace inherits from multiple parents the linear order of precedence (which namespace overrides which) is determined by the so-called C3 algorithm used to find the linear method resolution order for class hierarchies in Python and several other object oriented programming languages. The result of this should be fairly obvious for typical use of multiple inheritance in cylc suites, but for detailed documentation of how the algorithm works refer to the official Python documentation here: http://www.python.org/download/releases/2.3/mro/.
The inherit.multi.one example suite, listed here, makes use of multiple inheritance:
cylc get-config provides an easy way to check the result of inheritance in a suite. You can extract specific items, e.g.:
or use the --sparse option to print entire namespaces without obscuring the result with the dense runtime structure obtained from the root namespace:
The linear precedence order of ancestors is computed for each namespace using the C3 algorithm. Then any runtime items that are explicitly configured in the suite definition are “inherited” up the linearized hierachy for each task, starting at the root namespace: if a particular item is defined at multiple levels in the hiearchy, the level nearest the final task namespace takes precedence. Finally, root namespace defaults are applied for every item that has not been configured in the inheritance process (this is more efficient than carrying the full dense namespace structure through from root from the beginning).
The task execution environment contains suite and task identity variables provided by cylc, and user-defined environment variables. The environment is explicitly exported (by the task job script) prior to executing task command scripting (see Task Job Submission, Section 10).
Suite and task identity are exported first, so that user-defined variables can refer to them. Order of definition is preserved throughout so that variable assignment expressions can safely refer to previously defined variables.
Additionally, access to cylc itself is configured prior to the user-defined environment, so that variable assignment expressions can make use of cylc utility commands:
User Environment Variables A task’s user-defined environment results from its inherited [[[environment]]] sections:
This results in a task foo with SHAPE=circle, COLOR=blue, and TEXTURE=rough in its environment.
Overriding Environment Variables When you override inherited namespace items the original parent item definition is replaced by the new definition. This applies to all items including those in the environment sub-sections which, strictly speaking, are not “environment variables” until they are written, post inheritance processing, to the task job script that executes the associated task. Consequently, if you override an environment variable you cannot also access the original parent value:
The compressed variant of this, COLOR = dark-$COLOR, is also in error for the same reason. To achieve the desired result you must use a different name for the parent variable:
Suite And Task Identity Variables The task identity variables provided to tasks by cylc are:
And the suite identity variables are:
The variable $CYLC_SUITE_REG_PATH is just $CYLC_SUITE_REG_NAME (the hierarchical name under which the suite definition is registered in your suite database) translated into a directory path. This can be used when configuring suite logging directories and the like to put suite output in a directory tree that reflects the suite registration hierarchy (as opposed to the namespace hierarchy).
Some of these variables are also used by cylc task messaging commands in order to automatically target the right task proxy object in the right suite.
Suite Share And Task Work Directories See the variable listing above, and Sections A.4.1.11.2 and A.4.1.11.3.
Task command scripting is executed from within a work directory created on the fly, if necessary, by the task’s job script. In non-detaching tasks the work directory is automatically removed again if it is empty before the job script exits.
The share directory is also created on the fly, if necessary, by the job script. It is intended as a shared data area for multiple tasks on the same host, but as for any task runtime config item it can be specialized to particular tasks or groups of tasks.
The code for creating these directories, and removing empty work directories, can be seen by examining a task job script.
Other Cylc-Defined Environment Variables Initial and final cycle times, if supplied via the suite.rc file or the command line, are passed to task execution environments as:
Running tasks can use these to determine whether or not they are running in the first or final cycles.
Environment Variable Evaluation Variables in the task execution environment are not evaluated in the shell in which the suite is running prior to submitting the task. They are written in unevaluated form to the job script that is submitted by cylc to run the task (Section 10.1) and are therefore evaluated when the task begins executing under the task owner account on the task host. Thus $HOME, for instance, evaluates at run time to the home directory of task owner on the task host.
If a task declares an owner other than the suite owner and/or a host other than the suit host, e.g.:
cylc will execute the task on the declared host, by the configured job submission method, as the declared owner, using passwordless ssh.
A local task to run under another user account is treated as a remote task.
You may not need this functionality if you have a cross-platform resource manager, such as loadleveler, that allows you to submit a job locally to run on the remote host.
Remote host functionality, like all namespace settings, can be declared globally (in the root namespace) or per family, or for individual tasks. Use the global settings if all or most of your tasks need to run on the same remote host.
If cylc is not in your $PATH on the remote host you can use the “cylc directory” item to give remote tasks access to cylc. However, the remote job submission mechanism automatically sources .profile if it exists, so you can also set your PATH for cylc there. Similarly tasks can use $CYLC_SUITE_DEF_PATH to access suite files on the task host, and the suite bin directory is automatically added $PATH. If a remote suite definition directory is not specified the local (suite host) path will be assumed with the local home directory, if present, substituted with literal $HOME for evaluation on the task host.
Dynamic Host Selection Instead of hardwiring task host names into the suite definition you can specify a shell back-tick expression, as the value of the host config item, which executes an external command (it may be a script located in the suite bin directory) that writes a hostname to stdout.
Remote Task Log Directories Task stdout and stderr streams are written to log files in a suite-specific sub-directory of the suite run directory, as explained in Section 10.3. For remote tasks the same directory is used, but on the task host. Remote task log directories, like local ones, are created on the fly, if necessary, during job submission.
The visualization section of a suite definition is used to configure suite graphing, principally graph node (task) and edge (dependency arrow) style attributes. Tasks can be grouped for the purpose of applying common style attributes. See the suite.rc reference (Appendix A) for details.
Nested families from the namespace inheritance hierarchy, even if they are not used as family triggers in the graph, can be expanded or collapsed in suite graphs and in the suite control GUI’s text and graph views.
In the graph view ungraphed tasks, which includes the members of collapsed families, are automatically plotted as rectangular nodes to the right of the main graph if they are doing anything interesting (submitted, running, or failed).
Note that family relationships can be defined purely for visualization purposes - you can group tasks at root level in the inheritance hierarchy prior to defining real properties at higher levels.
Figure 32 illustrates successive expansion of nested task families in the namespaces example suite, which has the following namespace hierarchy:
Support for the Jinja2 template processor adds general variables, mathematical expressions, loop control structures, and conditional expressions to suite.rc files - which are automatically preprocessed to generate the final suite definition seen by cylc.
The need for Jinja2 processing must be declared with a hash-bang comment as the first line of the suite.rc file:
Potential uses for this include automatic generation of repeated groups of similar tasks and dependencies, and inclusion or exclusion of entire suite sections according to the value of a single flag. Consider a large complicated operational suite and several related parallel test suites with slightly different task content and structure (the parallel suites, for instance, might take certain large input files from the operation or the archive rather than downloading them again) - these can now be maintained as a single master suite definition that reconfigures itself according to the value of a flag variable indicating the intended use.
Template processing is the first thing done on parsing a suite definition so Jinja2 expressions can appear anywhere in the file (inside strings and namespace headings, for example).
Jinja2 is well documented at http://jinja.pocoo.org/docs, so here we just provide an example suite that uses it. The meaning of the embedded Jinja2 code should be reasonably self-evident to anyone familiar with standard programming techniques.
The jinja2.ensemble example, graphed in Figure 33, shows an ensemble of similar tasks generated using Jinja2:
Here is the generated suite definition, after Jinja2 processing:
And finally, the jinja2.cities example uses variables, includes or excludes special cleanup tasks according to the value of a logical flag, and it automatically generates all dependencies and family relationships for a group of tasks that is repeated for each city in the suite. To add a new city and associated tasks and dependencies simply add the city name to list at the top of the file. The suite is graphed, with the New York City task family expanded, in Figure 34.
This functionality is not provided by Jinja2 by default, but cylc automatically imports the user environment to the template in a dictionary structure called environ. A usage example:
This example is emphasizes that the environment is read on the suite host at the time the suite definition is parsed - it is not, for instance, read at task run time on the task host.
Jinja2 variable values can be modified by “filters”, using pipe notation. For example, the built-in trim filter strips leading and trailing white space from a string:
(See official Jinja2 documentation for available built-in filters.)
Cylc also supports custom Jinja2 filters. A custom filter is a single Python function in a source file with the same name as the function (plus “.py” extension) and stored in one of the following locations:
In the filter function argument list, the first argument is the variable value to be “filtered”, and subsequent arguments can be whatever is needed. Currently there is one custom filter called “pad” in the central cylc Jinja2 filter directory, for padding string values to some constant length with a fill character - useful for generating task names and related values in ensemble suites:
Associative arrays (dicts in Python) can be very useful. Here’s an example, from $CYLC_DIR/examples/jinja2/dict:
Here’s the result:
The values of Jinja2 variables can be passed in from the cylc command line rather than hardwired in the suite definition. Here’s an example, from $CYLC_DIR/examples/jinja2/defaults:
Here’s the result:
Note also that cylc view --set FIRST_TASKbob –jinja2 SUITE= will show the suite with the Jinja2 variables as set.
Warning: suites started with template variables set on the command line do not currently restart with the same settings - you have to set them again on the cylc restart command line.
Several special variables are used as placeholders in cylc suite definitions:
To use proper variables (c.f. programming languages) in suite definitions, see the Jinja2 template processor (Section 8.6).
It is sometimes convenient to omit certain tasks from the suite at runtime without actually deleting their definitions from the suite.
Defining [runtime] properties for tasks that do not appear in the suite graph results in verbose-mode validation warnings that the tasks are disabled. They cannot be used because the suite graph is what defines their dependencies and valid cycle times. Nevertheless, it is legal to leave these orphaned runtime sections in the suite definition because it allows you to temporarily remove tasks from the suite by simply commenting them out of the graph.
To omit a task from the suite at runtime but still leave it fully defined and available for use (by insertion or cylc submit) use one or both of [scheduling][[special task]] lists, include at start-up or exclude at start-up (documented in Sections A.3.5.8 and A.3.5.7). Then the graph still defines the validity of the tasks and their dependencies, but they are not actually inserted into the suite at start-up. Other tasks that depend on the omitted ones, if any, will have to wait on their insertion at a later time or otherwise be triggered manually.
Finally, with Jinja2 (Section 8.6) you can radically alter suite structure by including or excluding tasks from the [scheduling] and [runtime] sections according to the value of a single logical flag defined at the top of the suite.
A naked dummy task appears in the suite graph but has no explicit runtime configuration section. Such tasks automatically inherit the default “dummy task” configuration from the root namespace. This is very useful because it allows functional suites to be mocked up quickly for test and demonstration purposes by simply defining the graph. It is somewhat dangerous, however, because there is no way to distinguish an intentional naked dummy task from one generated by typographic error: misspelling a task name in the graph results in a new naked dummy task replacing the intended task in the affected trigger expression; and misspelling a task name in a runtime section heading results in the intended task becoming a dummy task itself (by divorcing it from its intended runtime config section).
To avoid this problem any dummy task used in a real suite should not be naked - i.e. it should have an explicit entry in under the runtime section of the suite definition, even if the section is empty. This results in exactly the same dummy task behaviour, via implicit inheritance from root, but it allows use of cylc validate --strict to catch errors in task names by failing the suite if any naked dummy tasks are detected.
This section lays out the minimal requirements on external commands, scripts, or executables invoked by cylc to carry out task processing.
Any existing command, script, or executable can function as a cylc task (or rather, perform the external processing that the task represents), with the following conditions excepions:
If these requirements are not met, see Tasks Requiring Modification For Cylc, Section 9.4.
The following suite runs a couple of external scripts that are not cylc-aware, but which meet the requirements above so that no special treatment is required at all:
Simple tasks can be entirely implemented within the suite.rc file because the task command scripting string can contain as many lines of code as you like.
You can, if you like, modify task scripts to send explanatory or progress messages to the suite as the task runs. For example, a task can send a priority critical message before aborting on error:
You can also use this syntax:
But not this:
You can also send warning messages, or general information:
This may be useful - any message received from a task is logged by cylc - but it is not a requirement. If error messages are not reported, for instance, detected task failures will still be registered, and task stdout and stderr logs can still be examined for evidence of what went wrong.
There are two main categories of tasks that require some modification to work with cylc: those with internal (pre-completion) outputs that other tasks need to trigger off, and those that spawn detaching internal processes prior to exiting before the spawned processing is finished. Additionally, any task that fails to indicate a fatal error by return non-zero exit status must be corrected.
The requirement to abort with non-zero exit status on error (which should be normal scripting practice in any case) allows the task job script to trap errors and send a cylc task failed message to alert the suite. You can use set -e to avoid writing explicit error checks for every operation:
See Section 9.3, Voluntary Messaging Modifications, for more examples of error detection and cylc messaging.
If a task has internal outputs that others can trigger off before it finishes, then it must report completion of those outputs with messages sent back to the suite at appropriate times. Output messages must be unique within the suite or else downstream tasks will trigger off whichever task happens to send the message first; they must exactly match the corresponding outputs registered for the task because cylc distinguishes between registered outputs that others can trigger off and general messages that are just logged; and for cycling tasks they must contain the cycle time in order to distinguish between the outputs of successive instances of the same task.
The “outputs” example is a self-contained suite that illustrates use of internal outputs:
Note that cycle time in the output message registration is expressed by a special placeholder variable (see Section 8.7) not by the corresponding environment variable. This is because registered output messages are held by task proxies, inside cylc, for comparison with incoming task messages; they are never interpreted by the shell and consequently may not contain environment variables. The actual messaging calls made by running tasks, on the other hand, can make use of variables in the task runtime environment.
You may be able to convert a detaching task to a non-detaching task very easily. If the detaching process is just a background shell process, for instance, run it in the foreground instead; for loadleveler the -s option prevents llsubmit from returning until the job has completed; for Sun Grid Engine, qsub -sync yes has the same effect. Section 10.4 shows how to override the command template used by cylc in order to customize job submission command options like this.
Tasks with processes that spawn jobs internally (e.g. to a batch queue scheduler or to another host) and then detach and exit without seeing the resulting processing through must arrange for the spawned processing to send its own “cylc task succeeded” or “cylc task failed” messages on completion, because the cylc-generated job script (Section 10.1) that otherwise arranges for automatic completion messaging cannot know when the task is really finished.
First check that you can’t easily “reconnect” the detaching internal processes, as described above in Section 9.4.3. If not, then start by disabling cylc’s automatic completion messaging:
Now, reporting success or failure is just a matter of calling the cylc messaging commands:
Bear in mind, however, that cylc messaging commands read environment variables that identify the calling task and the target suite, so if your job submission method does not automatically copy its parent environment you must arrange for these variables, at the least, to be propagated through to your spawned sub-jobs.
One way to handle this is to write a task wrapper that modifies a copy of the detaching native job scripts, on the fly, to insert completion messaging in the appropriate places, and other variables if necessary, before invoking the (now modified) native process. A significant advantage of this method is that you don’t need to permanently modify the model or its associated native scripting for cylc. Another is that you can configure the native job setup for a single test case (running it without cylc) and then have your custom wrapper modify the test case on the fly with suite, task, and cycle-specific parameters as required.
To make this easier, for tasks that declare manual completion messaging, cylc makes non user-defined environment scripting available in a single variable called $CYLC_SUITE_ENVIRONMENT that can be inserted into the aforementioned native task scripts prior to calling the cylc messaging commands.8
The detaching example suite contains a script model.sh that runs a pseudo-model executable as follows:
this is in turn executed by a script run-model.sh that detaches immediately after job submission (i.e. it exits before the model executable actually runs):
Note that your at scheduler daemon must be up if you want to test this suite.
Here’s a cylc suite to run this unruly model:
The suite invokes the task by means of the custom wrapper model-wrapper.sh which modifies, on the fly, a temporary copy of the model’s native job scripts as described above:
If you run this suite, or submit the model task alone with cylc submit, you’ll find that the usual job submission log files for task stdout and stderr end before the task is finished. To see the “model” output and the final task completion message (success or failure), examine the log files generated by the job submitted internally to the at scheduler (their location is determined by the $PREFIX variable in the suite.rc file).
It should not be difficult to adapt this example to real tasks with detaching internal job submission. You will probably also need to replace other parameters, such as model input and output filenames, with suite- and cycle-appropriate values, but exactly the same technique can be used: identify which job script needs to be modified and use text processing tools (such as the single line perl search-and-replace expressions above) to do the job.
Task Implementation (Section 9) describes what requirements a command, script, or program, must fulfill in order to function as a cylc task. This section explains how tasks are submitted by cylc when they are ready to run, and how to define new task job submission methods.
When a task is ready to run cylc generates a temporary task job script to configure the task’s execution environment and call its command scripting. The job script is the embodiment of all suite.rc runtime settings for the task. It is submitted to run by the job submission method configured for the task. Different tasks can have different job submission methods. Like other runtime properties, you can set a suite default job submission method and override it for specific tasks or families:
The actual command line used to submit the job script is written to stdout by cylc. In the following shell transcript we generate a job script for a task in the examples.QuickStart.c example suite and then examine it:
Here is the generated job script (note that some config items not used in this suite, such as task initial scripting , result in extra job script sections not shown here):
You can also generate a job script and print it directly to stdout with cylc jobscript SUITE TASKID.
Cylc has a number of built-in job submission methods. If these do not suite your needs Section 10.5 shows how to extend cylc with new user-defined job submission methods.
There are also two job submission methods intended for use in cylc development and testing: background_slow, in which the job submission subprocess itself takes ten seconds to complete before the task starts executing; and fail, in which the job submission subprocess itself always fails.
This job submission method runs tasks directly in a background shell.
This job submission method submits tasks to the rudimentary Unix at scheduler. The atd daemon must be running.
This job submission method submits tasks to loadleveler using the llsubmit command. Loadleveler directives can be provided in the suite.rc file:
These are written to the top of the task job script like this:
This job submission method submits tasks to PBS (or Torque) using the qsub command. PBS directives can be provided in the suite.rc file:
These are written to the top of the task job script like this:
This job submission method submits tasks to Sun Grid Engine using the qsub command. SGE directives can be provided in the suite.rc file:
These are written to the top of the task job script like this:
For loadleveler, pbs, and sge job submission, default directives are provided to set the job name and stdout and stderr file paths.
As shown in the example above, multiple entries for the same PBS or SGE directive option must be comma-separated on the same line, in the suite.rc file. Otherwise, repeating the option on another line will override the previous entry, not add to it. Also, the right-hand side must be quoted to hide the comma from the suite.rc parser (commas indicate list values, whereas directives are treated as singular).
As also shown in the example above, to get a naked option flag such as -cwd in SGE you must give a quoted blank space as the directive value in the suite.rc file.
When a task is ready to run cylc generates a filename root to be used for the task job script and log files. The filename containing the task name, cycle time (or integer tag), and a submit number that increments if the same task is re-triggered multiple times:
How the stdout and stderr streams are directed into these files depends on the job submission method. The background method just uses appropriate output redirection on the command line, as shown above. The loadleveler method writes appropriate directives to the job script that is submitted to loadleveler.
Cylc obviously has no control over the stdout and stderr output from tasks that do their own internal output management (e.g. tasks that submit internal jobs and direct the associated output to other files). For less internally complex tasks, however, the files referred to here will be complete task job logs.
Cylc’s suite control GUIs can display the task job logs (updating in real time for local tasks).
To change the form of the actual command used to submit a job you do not need to define a new job submission method; just override the command template in the relevant job submission sections of your suite.rc file:
As explained in the suite.rc reference (Appendix A), the template’s first %s will be substituted by the job file path and, where applicable a second and third %s will be substituted by the paths to the job stdout and stderr files.
Defining a new job submission method requires some minimal Python programming. You can derive (in the sense of object oriented programming inheritance) new methods from one of the existing ones, or directly from cylc’s job submission base class,
using the existing methods as examples. Most often this should merely be a matter of defining the command line used to execute the aforementioned job scripts and using the provided stdout and stderr file paths appropriately. For example, here is the entire class code for the background method:
Here is the at method:
And here is the pbs method:
The following user-defined job submission class, called qsub, overrides the built-in pbs class to change the directive prefix from #PBS to #QSUB:
To check that this works correctly save the new source file to qsub.py in one of the allowed locations (see just below), use it in a suite definition:
and generate a job script to see the resulting directives:
You new job submission class code should be saved to a file with the same name as the class (plus “.py” extension). It can reside in any of the following locations, depending on how generally useful the new method is and whether or not you have write-access to the cylc source tree:
Note that the form of the import statement at the top of the new user-defined Python module differs depending on whether or not the file is installed in the cylc source tree (see the comment at the top of the example file above).
This section may be incomplete - please see also the Quick Start Guide (Section 6), command documentation (Section C), and play with the example suites.
All interaction with a running suite by control commands, gcylc GUI instances, and messaging commands from running tasks, uses the Pyro network RPC (Remote Procedure Call) protocol. Each suite listens on its own network port; cylc starts grabbing ports at 7766 by default - this is configured in the site config file $CYLC_DIR/conf/site/site.rc, see cylc get-global-config --help.
However, if the Pyro network ports are blocked at your site you can get remote control commands and remote task messaging calls to re-invoke themselves on the suite host using passwordless ssh, so that the ultimate Pyro connection to the suite only occurs on the suite host.
All Pyro connections to a suite require passphrase authentication. The passphrase is just an arbitary single line of text. A secure MD5 checksum is passed across the network, not the raw passphrase itself. A random suite passhprase is generated in the suite definition directory when a suite is registered, but you can create your own as you wish.
Passphrases currently have to be installed manually in any user account, local or remote, that hosts tasks from the suite, or from which you want to interact with the suite without using the ssh remote communication method. Legal passphrase locations, in order of preference, are:
The locations under $HOME/.cylc are suitable for remote suite control accounts - in which case the suite definition directory may not be known or accessible. On remote task hosts the suite definition directory defaults to the same location relative to $HOME as on the suite host; or the remote location can be specified explicitly in the suite definition.
For ssh-based remote task messaging and suite control, ssh keys must be installed in the relevant accounts to allow passwordless ssh connections back to the suite host account. The suite passphrase is then only required on the suite host.
The Pyro connection method is the default because it is simpler and more efficient, but if your remote task host does not have the right network ports open, and the IT staff are reluctant to change that, then the ssh alternative is available so long as your local system and network configurations do not also prevent ssh connections back to the suite host - in that case see Section 11.1.4 below.
To use the ssh-based method:
It has come to our attention that some HPC facilities do not have any networking routing back out of the compute nodes, for security reasons and/or to avoid network chatter that could affect performance. Any kind of communication back to a cylc suite running on an external host would then presumably be impossible, but:
A restarted suite (see cylc restart --help) is initialized from a previous recorded suite state dump, so that it can carry on from wherever it got to before being shut down or killed.
Prior to cylc-5.0 any tasks recorded in the submitted or running states were automatically resubmitted on restarting, on the basis that they might not have completed while the suite was down and so should re-run just to be safe. Similarly, any tasks recorded as failed were automatically resubmitted on the basis that whatever caused them to fail might have been fixed while the suite was down.
Since cylc-5.0 we no longer assume anything on restarting - task proxies are now loaded in exactly their recorded states and it is up to users to intervene in restarted suites according to their knowledge of what happened to the real tasks at or after suite shutdown (e.g. by re-triggering task proxies shown in the running state if the corresponding real tasks are actually not still running).
As a suite runs its tasks (or rather the task proxies that represent the real tasks) may move through a number of defined states:
Finally, there is also a pseudo-state displayed by the gcylc graph view:
And a pseudo-state for task reset purposes:
All Pyro communication with a running suite requires knowing the network port on which the suite is listening. Running tasks know their own suite’s port number because the suite puts it in the task execution environment, but user commands must determine the port number somehow. Through to the 4.5.1 release, inclusive, cylc used port scanning to find the target port, but this could potentially cause delays on hosts with a large number of suites running at once. Consequently suites now write their own port number to $HOME/.cylc/ports/<SUITE> at start-up, and commands issued on the suite host read this file to get the port number. Commands issued on a remote host use passwordless ssh to the suite owner account to retrieve the port number. Alternatively, if passwordless ssh is not configured to the suite host the port number can be given on the command line with the --port option. If you accidentally delete a suite’s port file while the suite is still running, use cylc scan to determine the port number to use on the command line.
Cylc now handles task job submission in a dedicated worker thread so that submission of many remote tasks at once does not impact cylc’s performance or responsiveness.
Further, for maximum efficiency, job submissions are batched inside the worker thread: batch members are submitted in parallel, and all members must complete (the job submission process, that is, not the submitted task) before the next batch is handled. There is a configurable delay between batches to avoid swamping the host system in the event that hundreds of tasks become ready at the same time. The default batch size of 10 and delay of 15 seconds can be overridden in suite definitions:
Here a 120 task ensemble, for example, would be submitted in two batches of 50 followed by one of 20, with a 30 second delay between batches.
A connection timeout can be set in site/user config files (see Section 4.3) so that messaging commands cannot hang indefinitely if the suite is not responding (thie can be caused by suspending a suite with Ctrl-Z) thereby preventing the task from completing. The same can be done on the command line for other suite-connecting user commands, with the --pyro-timeout option.
Some cylc suites have the potential to generate too much activity at once by virtue of the fact that each task cycles independently constrained only by dependence on other tasks or by clock triggers. Quick-running tasks at the top of the dependency tree with no prerequisites and no clock-triggers (or when running far behind the clock) will spawn rapidly into the future if not constrained somehow. There are two issues to be aware of here: over-burdening task host resources by submitting too many tasks at once, and over-burdening cylc itself by letting the task pool become too big (when fast tasks spawn ahead of the pack cylc has to keep them around in the succeeded state until other tasks, which may depend on them, have caught up).
The runahead limit prevents the fastest tasks in a suite from getting too far ahead of the slowest ones. Cylc’s cycle-interleaving abilities make for generally efficient scheduling, but there is no great advantage in letting a few fast data retrieval tasks, say, get far ahead of the slower tasks because it is typically the tasks at the bottom of the dependency tree, which necessarily run last, that generate the final products.
A cycling task spawns its successor when it enters the submitted state or, for sequential tasks, when it finishes. If a newly spawned task’s cycle time is ahead of the oldest non-finished (succeeded or failed) task by more than the runahead limit it is put into the special runahead held state until other tasks catch up sufficiently.
In cylc-4.5.1 and earlier it was possible to stall a suite by setting the runahead limit too low; now however it is applied in such a way that it simply constrains the number of different cycles that can run at once. If the limit is set to the smallest interval between cycles, or less than that, just a single cycle will run at a time.
The default runahead limit is now set to twice the smallest of the cycling intervals of the suite’s cycling modules (i.e. the graph section headings). For a suite that has 1- and 24-hourly cylcing tasks the default limit will be 2 hours, so that two of the hourly cycles will run at once in between the 24-hourly cycles.
Succeeded and failed tasks are ignored when computing the runahead limit (but tasks that can’t run because they depend on a failed task are not ignored, of course).
Large suites could potentially swamp the task host hardware or external batch queueing system, depending on the chosen job submission method, by submitting too many tasks at once. Cylc’s internal queues prevent this by limiting the number of tasks, within defined groups, that are active (submitted or running) at once.
A queue is defined by a name; a limit, which is the maximum number of active tasks allowed for the queue; and a list of member tasks, which are assigned by name to the queue.
Queue configuration is done under the [scheduling] section of the suite.rc file, not as part of the runtime namespace hierarchy, because like dependencies queues constrain when a task runs rather than what runs after it is submitted. When runtime family relationships and queues do coincide you can assign task family members en masse to queues by using the family name, as shown in the example suite listing below.
By default every task is assigned to a default queue, which by default has a zero limit (interpreted by cylc as no limit). To use a single queue for the whole suite just set the default queue limit:
To use other queues just name each one, set the limit, and assign member tasks:
Any tasks not assigned to a particular queue will remain in the default queue. The queues example suite illustrates how queues work by running two task trees side by side (as seen in the graph GUI) each limited to 2 and 3 tasks respectively:
Note assignment of runtime task family members to queues using the family name.
See also Section A.4.1.7 in the Suite.rc Reference.
Tasks can be configured with a list of “retry delay” periods, in minutes, such that if a task fails it will go into a temporary retrying state and then automatically resubmit itself after the next specified delay period expires. A usage example is shown in the suite listed below under Suite And Task Event Handling, Section 11.8.
See also Sections A.2.7 and A.4.1.17 in the Suite.rc Reference.
Cylc can call nominated event handlers when certain suite or task events occur. This is intended to facilitate centralized alerting and automated handling of critical events. Event handlers can send an email or an SMS, call a pager, and so on; or intervene in the operation of their own suite using cylc commands. cylc [hook]email-suite and cylc [hook]email-task are example event handlers packaged with cylc.
Custom task event handlers can be located in the suite bin directory. They are passed the following arguments by cylc:
where EVENT is one of the following strings:
MESSAGE, if provided, describes what has happened, and TASKID identifies the task (NAME.CYCLE for cycling tasks).
The retry event occurs if a task fails and has any remaining retries configured (see Section 11.7). The event handler will be called as soon as the task fails, not after the retry delay period when it is resubmitted.
Note that event handlers are called by cylc itself, not by the running tasks so if you wish to pass them additional information via the environment you must use [cylc] → [[environment]], not task runtime environments.
Here is an example suite that tests the retry and failed events. The handler in this case simply echoes its command line arguments to suite stdout.
The cylc reload command, an experimental feature new in cylc-4.5.2, reloads the suite definition at run time. This allows: (a) changing task config such as command scripting or environment; (b) adding tasks to, or removing them from, the suite definition, at run time - without shutting the suite down and restarting it. (It is easy to shut down and restart cylc suites, but reloading may be useful if you don’t want to wait for long-running tasks to finish first).
Note that defined tasks can be already be added to or removed from a running suite with the ’cylc insert’ and ’cylc remove’ commands; the reload command allows addition and removal of task definitions. If a new task is definition is added (and used in the graph) you will still need to manually insert an instance of it (with a particular cycle time) into the running suite. If a task definition (and its use in the graph) is deleted, existing task proxies of the of the deleted type will run their course after the reload but new instances will not be spawned. Changes to a task definition will only take effect when the next task instance is spawned (existing instances will not be affected).
Some HPC facilities allow job preemption: the resource manager can kill or suspend running low priority jobs in order to make way for high priority jobs. The preempted jobs may then be automatically restarted by the resource manager, from the same point (if suspended) or requeued to run again from the start (if killed). If a running cylc task gets suspended or hard-killed (kill -9 <PID> is not a trappable signal so cylc cannot detect task failure in this case) and then later restarted, it will just appear to cylc as if it takes longer than normal to run. If the job is soft-killed the signal will be trapped by the task job script and a failure message sent, resulting in cylc putting the task into the failed state. When the preempted task restarts and sends its started message cylc would normally treat this as an error condition (a dead task is not supposed to be sending messages) - a warning will be logged and the task will remain in the failed state. However, if you know that preemption is possible on your system you can tell cylc that affected tasks should be resurrected from the dead, to carry on as normal if progress messages start coming in again after a failure:
To test this in any suite, manually kill a running task then, after cylc registers the task failed, resubmit the killed job manually by cutting-and-pasting the original job submission command from the suite stdout stream.
The cylc broadcast command overrides [runtime] settings in a running suite. This can be used to communicate information to downstream tasks by broadcasting environment variables (communication of information from one task to another normally takes place via the filesystem, i.e. the input/output file relationships embodied in inter-task dependencies). Variables (and any other runtime settings) may be broadcast to all subsequent tasks, or targetted specifically at a specific task, all subsequent tasks with a given name, or all tasks with a given cycle time; see broadcast command help for details.
Broadcast settings targetted at a specific task ID or cycle time expire and are forgotten as the suite moves on. Untargetted variables and those targetted at a task name persist throughout the suite run, even across restarts, unless manually cleared using the broadcast command - and so should be used sparingly.
When a suite is started with the cylc run command (cold or warm start) the cycle time at which it starts can be given on the command line or hardwired into the suite.rc file:
or,
An initial cycle time given on the command line will override one in the suite.rc file.
In the case of cold starts only the initial cycle time will also be passed through to task execution environments as $CYLC_SUITE_INITIAL_CYCLE_TIME. The intended use of this variable is to allow tasks to determine whether they are running in the initial cold-start cycle (when different behaviour may be required) or in a normal mid-run cycle. This is not done for warm starts because a warm start is really an implicit restart - it does not reference a particular previous suite state but it does assume that a previous cycle (for each task) has been run and completed entirely. It follows that in a warm start tasks are really in a normal mid-run cycle, and because no actual previous state is referenced $CYLC_SUITE_INITIAL_CYCLE_TIME gets the value None. After a cold-start, however, the value of the environment variable does persist across restarts because the original cold-start cycle time is stored in suite state dump files.
Since cylc-4.6.0 any cylc suite can run in live, simulation, or dummy mode. Prior to that release simulation mode was a hybrid mode that replaced real tasks with local dummy tasks. This allowed local simulation testing of any suite, to get the scheduling right without running real tasks, but running dummy tasks locally does not add much value over a pure simulation (in which no tasks are submitted at all) because all job submission configuration has to be ignored and most task job script sections have to be cut out to avoid any code that could potentially be specific to the intended task host. So at 4.6.0 we replaced this with a pure simulation mode (task proxies go through the running state automatically within cylc, and no dummy tasks are submitted to run) and a new dummy mode in which only the real task command scripting is dummied out - each dummy task is submitted exactly as the task it represents on the correct host and in the same execution environment. A successful dummy run confirms not only that the scheduling works correctly but also tests real job submission, communication from remote task hosts, and the real task job scripts (in which errors such as use of undefined variables will cause a task to fail).
The run mode, which defaults to live, is set on the command line (for run and restart):
but you can configure the suite to force a particular run mode,
This can be used, for example, for demo suites that necessarily run out of their original context; or to temporarily prevent accidental execution of expensive real tasks during suite development.
Dummy task command scripting just prints a message and sleeps for ten seconds by default, but you can override this behaviour for particular tasks or task groups if you like. Here’s how to make a task sleep for twenty seconds and then fail in dummy mode:
Finally, in simulation mode each task takes between 1 and 15 seconds to “run” by default, but you can also alter this for particular tasks or groups of tasks:
Note that to get a failed simulation or dummy mode task to succeed on re-triggering, just change the suite.rc file appropriately and reload the suite definition at run time with cylc reload SUITE before re-triggering the task.
Dummy mode is equivalent to commenting out each task’s command scripting to expose the default scripting.
In simulation and dummy mode cylc uses an accelerated clock with configurable rate and offset relative to the suite’s initial cycle time. This affects the trigger time of any clock-triggered tasks in the suite, and the length of time between cycles if simulating “caught up” operation (without this a six-hour cycling suite, for instance, would wait six hours between cycles when simulating caught-up operation, even though the simulated or dummy tasks run very quickly). By configuring the initial clock offset you can quickly simulate how suites catch up and transition from delayed to real time operation.
See Section A.2.10 for accelerated clock configuration settings.
The run mode is recorded in the suite state dump file. Cylc will not let you restart a non-live mode suite in live mode, or vice versa - any attempt to do the former would certainly be a mistake (because the simulation mode dummy tasks do not generate any of the real outputs depended on by downstream live tasks), and the latter, while feasible, would corrupt the live state dump by turning it over to simulation mode. The easiest way to test a live suite in simulation mode, if you don’t want to obliterate the current state dump by doing a cold or warm start (as opposed to a restart from the previous state) is to take a quick copy of the suite and run the copy in simulation mode. However, if you really want to run a live suite forward in simulation mode without copying it, do this:
Reference tests are finite-duration suite runs that abort with non-zero exit status if any of the following conditions occur (by default):
The default shutdown event handler for reference tests is cylc hook check-triggering which compares task triggering information (what triggers off what at run time) in the test run suite log to that from an earlier reference run, disregarding the timing and order of events - which can vary according to the external queueing conditions, runahead limit, and so on.
To prepare a reference log for a suite, run it with the --reference-log option, and manually verify the correctness of the reference run.
To reference test a suite, just run it (in dummy mode for the most comprehensive test without running real tasks) with the --reference-test option.
A battery of reference tests are (or will soon be) used to automatically test cylc before posting a new release version. They can also be used at cylc upgrade time to check that changes in cylc will not break your own complex suites - the triggering check will catch any bug that causes a task to run when it shouldn’t, for instance; and even in a dummy mode reference test the full task job script (sans real command scripting) has to execute successfully on the proper task host by the proper job submission method.
The reference test can be configured with the following settings:
If the default reference test is not sufficient for your needs, firstly note that you can override the default shutdown event handler, and secondly that the --reference-test option is merely a short cut to the following suite.rc settings which can also be set manually if you wish:
The following topics have yet to be documented in detail.
Until release 4.2.2 cylc had a “central suite database” that users could export to and import from, for sharing suites. It was essentially just a special instance of a user suite database, held under the cylc admin account and with associated export and import commands to copy suite definition directories to and from a central store, with the suite owner’s username as the first hierarchical name component. However, it was not on the network, and the suite store had to be writeable by all users and hence very insecure. Rather than develop a network server for better security and wider access, as was the original intention, it was decided to remove this functionality entirely for the following reasons:
We may in the future recommend particular tools that can be used for suite discovery, revision control, and so on, with cylc suites.
A suite can contain a small number of large, internally complex tasks; a large number of small, simple tasks; or anything in between. Cylc can easily handle a large number of tasks, however, so there are definite advantages to fine-graining:
It should be possible to rerun a task by simply resubmitting it for the same cycle time. In other words, failure at any point during execution of a task should not render a rerun impossible by corrupting the state of some internal-use file, or whatever. It is difficult to overstate the usefulness of being able to rerun the same task multiple times, either outside of the suite with cylc submit, or by retriggering it within the running suite, when debugging a problem.
If a warm-cycled model simply overwrites its restart files in each run, the only cycle that can subsequently run is the next one. This is dangerous because if, accidentally or otherwise, the task runs for the wrong cycle time, its restart files will be corrupted such that the correct cycle can no longer run (probably necessitating a cold-start). Instead, consider organising restart files by cycle time, through a file or directory naming convention, and keep them in a simple rolling archive (cylc’s filename templating and housekeeping utilities can easily do this for you). Then, given availability of external inputs, you can easily rerun the task for any cycle still in the restart archive.
Cylc does not require that successive instances of the same task run sequentially. In order to task advantage of this and achieve maximum functional parallelism whenever the opportunity arises (usually when catching up from a delay) you should ensure that tasks that in principle do not depend on their own previous instances (the vast majority of tasks in most suites, in fact) do not do so in practice. In other words, they should be able to run as soon as their prerequisites are satisfied regardless of whether or not their predecessors have finished yet. This generally just means ensuring that all file I/O contains the generating task’s cycle time in the file or directory name so that there is no interference between successive instances. If this is difficult to achieve in particular cases, however, you can declare the offending tasks to be sequential.
Having all filenames, or perhaps the names of their containing directories, stamped with the cycle time of the generating task greatly aids in managing suite disk usage, both for archiving and cleanup. It also enables the aforementioned task rerunnability recommendation by avoiding overwrite of important files from one cycle to the next. Cylc has powerful utilities for cycle time offset filename templating and housekeeping.
The command line utility program cylc [util] cycletime computes offsets (in hours, days, months, and years) from a given or current (in the environment) cycle time, and optionally inserts the resulting computed cycle time, or components of it, into a given template string containing “YYYY” as a placeholder for the year value, “MM” for month, and so on. This can be used in the suite.rc environment or command scripting sections, or in task implementation scripting, to generate filenames containing the current cycle time (or some offset from it) for use by tasks.
See cylc [util] cycletime --help for examples.
Dependencies between tasks usually, though not always, take the form of files generated by one task that are used by other tasks. It is possible to manage these files across a suite without hard wiring I/O locations and therefore compromising suite flexibility and portability.
For small suites you may be able to have all tasks read and write from a common workspace, thereby avoiding the need to move common files around. You should be able to define the workspace location once in the suite.rc file rather than hard wiring it into the task implementations.
Tasks can be added to a suite to move files from A’s output directory to B’s input directory, and so on. Many connector tasks may be able to call the same file transfer script or command, with task-dependendent input parameters defined in the suite.rc file.
Whether or not your suite uses a single common workspace, passing common I/O paths to tasks via variables defined once in the suite.rc file should allow you to avoid using connector tasks at all, except where it is necessary to transfer files between machines, or similar.
If your suite contains multiple logically distinct tasks that actually have similar functionality (e.g. for moving files around, or for generating similar products from the output of several similar models) have the corresponding cylc tasks all call the same command, script, or executable - just provide different input parameters via the task command scripting and/or execution environment, in the suite.rc file.
If every task in a suite is configured to put its output under $HOME (i.e. the environment variable, literally, not the explicit path to your home directory; and similarly for temporary directories, etc.) then other users will be able to copy the suite and run it immediately, after merely ensuring that any external input files are in the right place.
For the ultimate in portability, construct suites in which all task I/O paths are dynamically configured to be user and suite (registration) specific, e.g.
(these variables are automatically exported to the task execution environment by cylc - see Task Execution Environment, Section 8.4.7). Then you can run multiple instances of the suite at once (even under the same user account) without changing anything, and they will not interfere with each other.
You can test changes to a portable suite safely by making a quick copy of it in a temporary directory, then modifying and running the test copy without fear of corrupting the output directories, suite logs, and suite state, of the original.
Where possible, no task should rely on the action of another task, except for the prerequisites embodied in the suite dependency graph that it has no choice but to depend on. If this rule is followed, your suite will be as flexible as possible in terms of being able to run single tasks, or subsets of the suite, whilst debugging or developing new features.9 For example, every task should create its own output directories if they do not already exist, instead of assuming their existence due to the action of some other task; then you will be able to run single tasks without having to manually create output directories first.
The only compulsory content of a cylc suite definition directory is the suite.rc file (and you’ll almost certainly have a suite bin sub-directory too). However, you can store whatever you like in a suite definition directory;10 other files there will be ignored by cylc but suite tasks can access them via the $CYLC_SUITE_DEF_PATH variable that cylc automatically exports into the task execution environment. Disk space is cheap - if all programs, ancillary files, control files (etc.) required by the suite are stored in the suite definition directory instead of having the suite reference external build directories (etc.), you can turn the directory into a revision control repository and be virtually assured of the ability to exactly reproduce earlier versions, regardless of suite complexity.
Correct scheduling is not equivalent to “orderly generation of products by cycle time”. Under cylc, a product generation task will trigger as soon as its prerequisites are satisfied (i.e. when its input files are ready, generally) regardless of whether other tasks with the same cycle time have finished or have yet to run. If your product delivery or presentation system demands that all products for one cycle time are uploaded (or whatever) before any from the next cycle, then be aware that this may be quite inefficient if your suite is ever faced with catching up from a significant delay or running over historical data.
If you must, however, you can introduce artificial dependencies into your suite to ensure that the final products never arrive out of sequence. One way of doing this would be to have a final “product upload” task that depends on completion of all the real product generation tasks at the same cycle time, and then declare it to be sequential.
All tasks in a cylc suite know their own private cycle time, but most don’t care about the wall clock time - they just run when their prerequisites are satisfied. The exception to this is clock-triggered tasks, which wait on a wall clock time expressed as an offset from their own cycle time, in addition to any other prerequisites. The usual purpose of these tasks is to retrieve real time data from the external world, triggering at roughly the expected time of availability of the data. Triggering the task at the right time is up to cylc, but the task itself should go into a check-and-wait loop in case the data is delayed; only on successful detection or retrieval should the task report success and then exit (or perhaps report failure and then exit if the data has not arrived by some cutoff time).
Cylc suites, without modification, can handle real time and delayed operation equally well.
In real time operation clock-triggered tasks constrain the behaviour of the whole suite, or at least of all tasks downstream of them in the dependency graph.
In delayed operation (whether due to an actual delay in an operational suite or because you’re running an historical trial) clock-triggered tasks will not constrain the suite at all, and cylc’s cycle interleaving abilities come to the fore, because their trigger times have already passed. But if a clock-triggered task happens to catch up to the wall clock, it will automatically wait again. In this way a cylc suite naturally and seamlessly transitions between delayed and real time operation as required.
This appendix documents legal content of raw cylc suite.rc files. Many items have sensible default values, and most suites may only need to explicitly configure a few of them.
In addition to the configuration items described below, Jinja2 expressions can also be embedded to programmatically generate the final suite definition seen by cylc. Use of Jinja2 is documented in Section 8.6.
See also Suite Definition - Suite.rc Overview (Section 8.2) for a descriptive overview of suite.rc files.
The only top level configuration items at present are the suite title and description.
A single line description of the suite. It is displayed in the db viewer window and can be retrieved at run time with the cylc show command.
A multi-line description of the suite. It can be retrieved by the db viewer right-click menu, or at run time with the cylc show command.
This section is for suite configuration that is not specifically task-related.
If this item is set cylc will abort if the suite is not started in the specified mode. This can be used for demo suites that have to be run in simulation mode, for example, because they have been taken out of their normal operational context; or to prevent accidental submission of expensive real tasks during suite development.
Cylc runs off the suite host’s system clock by default. This item allows you to run the suite in UTC even if the system clock is set to local time. Clock-triggered tasks will trigger when the current UTC time is equal to their cycle time plus offset; other time values used, reported, or logged by cylc will also be in UTC.
Cylc does not normally abort if tasks fail, but if this item is turned on it will abort with exit status 1 if any task fails.
If this is turned on cylc will write the resolved dependencies of each task to the suite log as it becomes ready to run (a list of the IDs of the tasks that actually satisfied its prerequisites at run time). Mainly used for cylc testing and development.
Tasks ready to submit are now queued for processing in a background worker thread, so submitting a lot of tasks at once does not hold cylc back. In the job submission thread tasks are are batched, with members of each batch being submitted in parallel. Batches are processed serially, with a delay between batches, to avoid swamping the host system with too many simultaneous job submissions.
The time required for a single task’s job submission to complete typically depends on whether it is a remote task (for which an ssh connection must be established and used) and whether dynamic host selection is used (see A.4.1.16.1 (a dynamic host selection command runs as part of the job submission command). The time taken for a batch of parallel job submissions to complete will be roughly the duration of the slowest member process.
[cylc] → [job submission] → batch size The maximum number of tasks to be submitted in a single batch, in the job submission thread. Cylc waits for all batch member job-submissions to complete before proceeding to the next batch.
[cylc] → [job submission] → delay between batches It may cause a problem for some batch queue schedulers to submit too many jobs at once, so cylc allows a configurable delay between job submission batches.
Task event handlers are queued to a background worker thread that limits the number of event handlers that can run at once, as for task job submissions (above), because of the potential for large suites to swamp the suite host with event handlers. Event handlers are currently executed directly as sub-processes (i.e. unlike task job submission you can’t specify the “job submission method” to use).
Note that suite event handlers are currently executed directly in the main scheduler thread, not queued to the task event handler worker thread.
[cylc] → [event handler execution] → batch size The maximum number of event handlers to be executed in parallel in the worker thread. Cylc waits for all batch members to complete before proceeding on to the next batch.
[cylc] → [event handler execution] → delay between batches A configurable delay between batches of task event handlers.
Cylc has internal “hooks” to which you can attach handlers that are called whenever certain events occur. This section is for configuring suite event hooks. See Section A.4.1.17 for task event hooks.
Event handlers can send an email or an SMS, call a pager, and so on; or intervene in the operation of their own suite using cylc commands. The command cylc [hook] email-suite is a ready-made suite event handler.
Custom suite event handlers can be located in the suite bin directory, in which case you will not need to modify your $PATH to ensure they are found. They are called by cylc with the following arguments:
EVENT is the event name (see below), SUITE is the suite name, and MESSAGE, if provided, describes what has happened.
Note that event handlers are called by cylc itself so if you wish to pass additional information to them via the environment you must use [cylc] → [[environment]], not task runtime environments (suite-level variables - $CYLC_SUITE_INITIAL_CYCLE_TIME etc. - are exported into the cylc environment, however).
[cylc] → [[event hooks]] → EVENT handler The handler to call when the suite event EVENT occurs. Repeat this section for each of the following suite events that you wish to handle:
[cylc] → [[event hooks]] → timeout If a timeout is set and the timeout event is handled, the event handler will be called if the suite times out before it finishes. The timer is set initially at suite start up.
[cylc] → [[event hooks]] → reset timer If True (the default) the suite timer will continually reset after any task changes state, so you can time out after some interval since the last activity occured rather than on absolute suite execution time.
[cylc] → [[event hooks]] → abort on timeout If a suite timer is set (above) this will cause the suite to abort with error status if the suite times out while still running.
[cylc] → [[event hooks]] → abort if EVENT handler fails Cylc does not normally care whether an event handler succeeds or fails, but if this is turned on the EVENT handler will be executed in the foreground (which will block the suite while it is running) and the suite will abort if the handler fails.
The cylc lockserver brokers suite and task locks on the network (these are somewhat analagous to traditional local lock files). It prevents multiple instances of a suite or task from being invoked at the same time (via scheduler instances or cylc submit).
See cylc lockserver --help for how to run the lockserver, and cylc lockclient --help for occasional manual lock management requirements.
[cylc] → [[lockserver]] → enable The lockserver is currently disabled by default. It is intended mainly for operational use.
[cylc] → [[lockserver]] → simultaneous instances By default the lockserver prevents multiple simultaneous instances of a suite from running even under different registered names. But allowing this may be desirable if the I/O paths of every task in the suite are dynamically configured to be suite specific (and similarly for the suite state dump and logging directories, by using suite identity variables in their directory paths). Note that the lockserver cannot protect you from running multiple distinct copies of a suite simultaneously.
Variables defined here are exported into the environment in which cylc itself runs. They are then available to local processes spawned directly by cylc. Any variables read by task event handlers must be defined here, for instance, because event handlers are executed directly by cylc, not by running tasks. And similarly the command lines issued by cylc to invoke event handlers or to submit task job scripts could, in principle, make use of environment variables defined here.
[cylc] → [[environment]] → __VARIABLE__ Replace __VARIABLE__ with any number of environment variable assignment expressions. Values may refer to other local environment variables (order of definition is preserved) and are not evaluated or manipulated by cylc, so any variable assignment expression that is legal in the shell in which cylc is running can be used (but see the warning above on variable expansions, which will not be evaluated). White space around the ‘=’ is allowed (as far as cylc’s suite.rc parser is concerned these are normal configuration items).
Accelerated clock settings, used to speed up the wait between cycles in the simulation and dummy run modes.
[cylc] → [[accelerated clock]] → disable Disabling the accelerated clock makes the suite (and its log time stamps etc.) run on real time. Note that if the suite has clock-triggered tasks that catch up to the wall clock, the interval between cycles will also be in real time - e.g. six hours for a six hourly cycle.
[cylc] → [[accelerated clock]] → rate The rate at which the accelerated clock runs in real seconds per simulated hour.
[cylc] → [[accelerated clock]] → offset The clock offset determines the initial time on the accelerated clock, at suite startup, relative to the initial cycle time. An offset of 0 simulates real time operation; greater offsets simulate catch up from a delay and subsequent transition to real time operation.
Reference tests are finite-duration suite runs that abort with non-zero exit status if cylc fails, if any task fails, if the suite times out, or if a shutdown event handler that (by default) compares the test run with a reference run reports failure. See Automated Reference Test Suites, Section 11.14.
[cylc] → [[reference test]] → suite shutdown event handler A shutdown event handler that should compare the test run with the reference run, exiting with zero exit status only if the test run verifies.
As for any event handler, the full path can be ommited if the script is located somewhere in $PATH or in the suite bin directory.
[cylc] → [[reference test]] → required run mode If your reference test is only valid for a particular run mode, this setting will cause cylc to abort if a reference test is attempted in another run mode.
[cylc] → [[reference test]] → allow task failures A reference test run will abort immediately if any task fails, unless this item is set, or a list of expected task failures is provided (below).
[cylc] → [[reference test]] → expected task failures A reference test run will abort immediately if any task fails, unless allow task failures is set (above) or the failed task is found in a list IDs of tasks that are expected to fail.
[cylc] → [[reference test]] → live mode suite timeout The timeout value in minutes after which the test run should be aborted if it has not finished, in live mode. Test runs cannot be done in live mode unless you define a value for this item, because it is not possible to arrive at a sensible default for all suites.
[cylc] → [[reference test]] → simulation mode suite timeout The timeout value in minutes after which the test run should be aborted if it has not finished, in simulation mode. Test runs cannot be done in simulation mode unless you define a value for this item, because it is not possible to arrive at a sensible default for all suites.
[cylc] → [[reference test]] → dummy mode suite timeout The timeout value in minutes after which the test run should be aborted if it has not finished, in dummy mode. Test runs cannot be done in dummy mode unless you define a value for this item, because it is not possible to arrive at a sensible default for all suites.
This section allows cylc to determine when tasks are ready to run.
At startup each cycling task (unless specifically excluded under [special tasks]) will be inserted into the suite with this cycle time, or with the closest subsequent valid cycle time for the task. Note that whether or not cold-start tasks, specified under [special tasks], are inserted, and in what state they are inserted, depends on the start up method - cold, warm, or raw. If this item is provided you can override it on the command line or in the gcylc suite start panel.
Cycling tasks are held once they pass the final cycle time, if one is specified. Once all tasks have achieved this state the suite will shut down. If this item is provided you can override it on the command line or in the gcylc suite start panel.
The suite runahead limit prevents the fastest tasks in a suite from getting too far ahead of the slowest ones, as documented in Section 11.6.1. Tasks exceeding the limit are put into a special runahead held state until slower tasks have caught up sufficiently.
Configuration of internal queues, by which the number of simultaneously active tasks (submitted or running) can be limited, per queue. By default a single queue called default is defined, with all tasks assigned to it and no limit. To use a single queue for the whole suite just set the limit on the default queue as required. See also Section 11.6.2.
[scheduling] → [[queues]] → [[[__QUEUE__]]] Section heading for configuration of a single queue. Replace __QUEUE__ with a queue name, and repeat the section as required.
[scheduling] → [[queues]] → [[[__QUEUE__]]] → limit The maximum number of active tasks allowed at any one time, for this queue.
[scheduling] → [[queues]] → [[[__QUEUE__]]] → members A list of member tasks, or task family names, to assign to this queue (assigned tasks will automatically be removed from the default queue).
This section identifies any tasks with special behaviour. By default (i.e. non “special” behaviour) tasks submit (or queue) as soon as their prerequisites are satisfied, and they spawn a successor at the next valid cycle time for the task as soon as they enter the submitted state11
[scheduling] → [[special tasks]] → clock-triggered Clock-triggered tasks wait on a wall clock time specified as an offset in hours relative to their own cycle time, in addition to any dependence they have on other tasks. Generally speaking, only tasks that wait on external real time data need to be clock-triggered. Note that in computing the trigger time the full wall clock time and cycle time are compared, not just hours and minutes of the day, so when running a suite in catchup/delayed operation, or over historical periods, clock-triggered tasks will not constrain the suite at all until they catch up to the wall clock.
[scheduling] → [[special tasks]] → start-up Start-up tasks are one-off tasks (they do not spawn a successor) that only run in the first cycle (and only in a cold-start) and any dependence on them is ignored in subsequent cycles. They can be used to prepare a suite workspace, for example, before other tasks run. Start-up tasks cannot appear in conditional trigger expressions with normal cycling tasks, because the meaning of the conditional expression becomes undefined in subsequent cycles.
[scheduling] → [[special tasks]] → cold-start A cold-start task is one-off task used to satisfy the dependence of an associated task with the same cycle time, on outputs from a previous cycle - when those outputs are not available. The primary use for this is to cold-start a warm-cycled forecast model that normally depends on restart files (e.g. model background fields) generated by its previous forecast, when there is no previous forecast. This is required when cold-starting the suite, but cold-start tasks can also be inserted into a running suite to restart a model that has had to skip some cycles after running into problems. Cold-start tasks can invoke real cold-start processes, or they can just be dummy tasks that represent some external process that has to be completed before the suite is started. Unlike start-up tasks, dependence on cold-start tasks is preseverved in subsequent cycles so they must typically be used in OR’d conditional expressions to avoid holding up the suite.
[scheduling] → [[special tasks]] → sequential By default, a task spawns a successor as soon as it is submitted to run so that successive instances of the same task can run in parallel if the opportunity arises (i.e. if their prerequisites happen to be satisfied before their predecessor has finished). Sequential tasks, however, will not spawn a successor until they have finished successfully. This should be used for (a) tasks that cannot run in parallel with their own previous instances because they would somehow interfere with each other (use cycle time in all I/O paths to avoid this); and (b) warm cycled forecast models that write out restart files for multiple cycles ahead (exception: see “explicit restart outputs” below).12
[scheduling] → [[special tasks]] → one-off Synchronous one-off tasks have an associated cycle time but do not spawn a successor. Synchronous start-up and cold-start tasks are automatically one-off tasks and do not need to be listed here. Dependence on one-off tasks is not restricted to the first cycle.
[scheduling] → [[special tasks]] → explicit restart outputs This is only required in the event that you need a warm cycled forecast model to start at the instant its restart files are ready (if other prerequisites are satisfied) even if its previous instance has not finished yet. If so, the model task has to depend on special output messages emitted by the previous instance as soon as its restart files are ready, instead of just on the previous instance finishing. Tasks in this category must define special restart output messages containing the word “restart”, in [runtime] → [[TASK]] → [[[outputs]]] - see Section 9.4.2.
[scheduling] → [[special tasks]] → exclude at start-up Any task listed here will be excluded from the initial task pool (this goes for suite restarts too). If an inclusion list is also specified, the initial pool will contain only included tasks that have not been excluded. Excluded tasks can still be inserted at run time. Other tasks may still depend on excluded tasks if they have not been removed from the suite dependency graph, in which case some manual triggering, or insertion of excluded tasks, may be required.
[scheduling] → [[special tasks]] → include at start-up If this list is not empty, any task not listed in it will be excluded from the initial task pool (this goes for suite restarts too). If an exclusion list is also specified, the initial pool will contain only included tasks that have not been excluded. Excluded tasks can still be inserted at run time. Other tasks may still depend on excluded tasks if they have not been removed from the suite dependency graph, in which case some manual triggering, or insertion of excluded tasks, may be required.
The suite dependency graph is defined under this section. You can plot the dependency graph as you work on it, with cylc graph or by right clicking on the suite in the db viewer. See also Section 8.3.
[scheduling] → [[dependencies]] → graph The dependency graph for any one-off asynchronous (non-cycling) tasks in the suite goes here. This can be used to construct a suite of one-off tasks (e.g. build jobs and related processing) that just completes and then exits, or an initial suite section that completes prior to the cycling tasks starting (if you make the first cycling tasks depend on the last one-off ones). But note that synchronous start-up tasks can also be used for the latter purpose. See Section A.3.6.2.1 below for graph string syntax, and Section 8.3.
[scheduling] → [[dependencies]] → [[[__VALIDITY__]]] __VALIDITY__ section headings define the sequence of cycle times for which the subsequent graph section is valid. For cycling tasks use a comma-separated list of integer hours, 0 ≤ H ≤ 23 for the original hours-of-the-day cycling, or reference a particular stepped daily, monthly, or yearly cycling module:
For repeating asynchronous tasks put ‘ASYNCID:pattern’ in the section heading, where pattern is a regular expression that matches an asynchronous task ID:
See Section 8.3.3, Graph Types for the meaning of the stepped cycler arguments, how multiple graph sections combine within a single suite, and so on.
[scheduling] → [[dependencies]] → [[[__VALIDITY__]]] → graph The dependency graph for the specified validity section (described just above) goes here. Syntax examples follow; see also Sections 8.3 (Configuring Scheduling) and 8.3.4 (Trigger Types).
[scheduling] → [[dependencies]] → [[[__VALIDITY__]]] → daemon For [[[ASYNCID:pattern]]] validity sections only, list asynchronous daemon tasks by name. This item is located here rather than under [scheduling] → [[special tasks]] because a damon task is associated with a particular asynchronous ID.
This section is used to specify how, where, and what to execute when tasks are ready to run. Common configuration can be factored out in a multiple-inheritance hierarchy of runtime namespaces that culminates in the tasks of the suite. Order of precedence is determined by the C3 linearization algorithm as used to find the method resolution order in Python language class hiearchies. For details and examples see Section 8.4, Runtime Properties.
Replace __NAME__ with a namespace name, or a comma separated list of names, and repeat as needed to define all tasks in the suite. Names may contain letters, digits, underscores, and hyphens. A namespace represents a group or family of tasks if other namespaces inherit from it, or a task if no others inherit from it.
If multiple names are listed the subsequent settings apply to each.
All namespaces inherit initially from root, which can be explicitly configured to provide or override default settings for all tasks in the suite.
[runtime] → [[__NAME__]] → inherit A list of the immediate parent(s) this namespace inherits from. If no parents are listed root is assumed.
[runtime] → [[__NAME__]] → title A single line description of this namespace. It is displayed by the cylc list command and can be retrieved from running tasks with the cylc show command.
[runtime] → [[__NAME__]] → description A multi-line description of this namespace, retrievable from running tasks with the cylc show command.
[runtime] → [[__NAME__]] → initial scripting Initial scripting is executed at the top of the task job script just before the cylc task started message call is made, and before the task execution environment is configured - so it does not have access to any suite or task environment variables. The original intention was to allow remote tasks to source login scripts before calling the first cylc command, e.g. to set $PYTHONPATH if Pyro has been installed locally. Note however that the remote task invocation mechanism now automatically sources both /etc/profile and $HOME/.profile if they exist. For other uses pre-command scripting should be used if possible because it can has access to the task execution environment.
[runtime] → [[__NAME__]] → environment scripting Environment scripting is inserted into the task job script between the cylc-defined environment (suite and task identity, etc.) and the user-defined task runtime environment - i.e. it has access to the cylc environment, and the task environment has access to the results of this scripting.
[runtime] → [[__NAME__]] → command scripting The scripting to execute when the associated task is ready to run - this can be a single command or multiple lines of scripting.
[runtime] → [[__NAME__]] → retry delays A list of time intervals in minutes, after which to resubmit the task if it reports failure. The variable $CYLC_TASK_TRY_NUMBER in the task execution environment is incremented each time, starting from 1 for the original try; this can be used to vary task behavior according to the try number.
[runtime] → [[__NAME__]] → pre-command scripting Scripting to be executed immediately before the command scripting. This would typically be used to add scripting to every task in a family (for individual tasks you could just incorporate the extra commands into the main command scripting). See also post-command scripting, below.
[runtime] → [[__NAME__]] → post-command scripting Scripting to be executed immediately after the command scripting. This would typically be used to add scripting to every task in a family (for individual tasks you could just incorporate the extra commands into the main command scripting). See also pre-command scripting, above.
[runtime] → [[__NAME__]] → manual completion If a task’s initiating process detaches and exits before task processing is finished then cylc cannot arrange for the task to automatically signal when it has succeeded or failed. In such cases you must use this configuration item to tell cylc not to arrange for automatic completion messaging, and insert some minimal completion messaging yourself in appropriate places in the task implementation (see Section 9.4.4).
[runtime] → [[__NAME__]] → enable resurrection If a message is received from a failed task cylc will normally treat this as an error condition, issue a warning, and leave the task in the “failed” state. But if “enable resurrection” is switched on failed tasks can come back from the dead: if the same task job script is executed again cylc will put the task back into the running state and continue as normal when the started message is received. This can be used to handle HPC-style job preemption wherein a resource manager may kill a running task and reschedule it to run again later, to make way for a job with higher immediate priority. See also Section 11.10, Handling Job Preemption
[runtime] → [[__NAME__]] → log directory This is where task job scripts, and the stdout and stderr logs for local tasks, are written. The directory path may contain environment variables, including suite identity variables to make the path suite-specific (as the default value does) but it may not contain task identity variables such as $CYLC_TASK_NAME and $CYLC_TASK_CYCLE_TIME or any variables defined in the task environment section - the directory has to be created by cylc before the task runs so the directory creation process does not see the task execution environment. The job script filename is constructed, just before job submission, from the task ID and seconds since epoch, and then .out and .err are appended to construct the stdout and stderr log names, respectively. These filenames are thus unique even if a task gets retriggered and yet will be correctly time ordered if the log directory is listed. The filenames are also recorded by the task proxies for access via cylc commands and the suite control GUIs.
[runtime] → [[__NAME__]] → work directory Task command scripting is executed from within a work directory created on the fly, if necessary, by the task’s job script. In non-detaching tasks the work directory is automatically removed again if it is empty before the job script exits. The work directory can be accessed by tasks via the environment variable $CYLC_TASK_WORK_PATH.
[runtime] → [[__NAME__]] → share directory Like task work directories (above) this directory is created on the fly, if necessary, by the job script. It is intended as a shared data area for multiple tasks on the same host, but as for any task runtime config item it can be specialized to particular tasks or groups of tasks. It can be accessed by tasks at run time via the environment variable $CYLC_SUITE_SHARE_PATH.
[runtime] → [[__NAME__]] → [[[dummy mode]]] Dummy mode configuration.
[runtime] → [[__NAME__]] → [[[dummy mode]]] → command scripting The scripting to execute when the associated task is ready to run, in dummy mode - this can be a single command or a multiple lines of scripting.
[runtime] → [[__NAME__]] → [[[dummy mode]]] → disable pre-command scripting This disables pre-command scripting, is likely to contain code specific to the real task, in dummy mode.
[runtime] → [[__NAME__]] → [[[dummy mode]]] → disable post-command scripting This disables post-command scripting, which is likely to contain code specific to the real task, in dummy mode.
[runtime] → [[__NAME__]] → [[[simulation mode]]] Simulation mode configuration.
[runtime] → [[__NAME__]] → [[[simulation mode]]] → run time range This defines an interval [min,max) (seconds) from within which the the simulation mode task run length will be randomly chosen.
[runtime] → [[__NAME__]] → [[[job submission]]] This section configures the means by which cylc submits task job scripts to run.
[runtime] → [[__NAME__]] → [[[job submission]]] → method See Task Job Submission (Section 10) for how job submission works, and how to define new methods. Cylc has a number of built in job submission methods:
[runtime] → [[__NAME__]] → [[[job submission]]] → command template This allows you to override the actual command used by the chosen job submission method. The template’s first %s will be substituted by the job file path. Where applicable the second and third %s will be substituted by the paths to the job stdout and stderr files.
[runtime] → [[__NAME__]] → [[[job submission]]] → shell This is the shell used to interpret the job script submitted by cylc when a task is ready to run. It has no bearing on the shell used in task implementations. Command scripting and suite environment variable assignment expressions must be valid for this shell. The latter is currently hardwired into cylc as export item=value - valid for both bash and ksh because value is entirely user-defined - but cylc would have to be modified slightly to allow use of the C shell.
[runtime] → [[__NAME__]] → [[[remote]]] Configur host and username, for tasks that do not run on the suite host. Cylc will use passwordless ssh to submit the task by the configured job submission method. Cylc must be installed on remote task hosts, but of the external software dependencies only Pyro is required there (actually, not even that if ssh messaging is used; see below). Passwordless ssh must be configured between the local suite owner on the suite host, and the task owner on the remote task host.
[runtime] → [[__NAME__]] → [[[remote]]] → host The remote host for this namespace. This can be a static hostname or a command that prints a suitable hostname to stdout. Host selection commands are executed just prior to job submission. The host (static or dynamic) may have an entry in the cylc site or user config file to specify parameters such as the location of cylc on the remote machine; if not, the corresponding local settings (on the suite host) will be assumed to apply on the remote host.
[runtime] → [[__NAME__]] → [[[remote]]] → owner The task owner username. This is (only) used in the passwordless ssh command line invoked by cylc to submit the remote task (consequently it may be defined using local environment variables (i.e. the shell in which cylc runs, and [cylc] → [[environment]]).
[runtime] → [[__NAME__]] → [[[remote]]] → suite definition directory The path to the suite definition directory on the remote host, needed if remote tasks require access to files stored there (via $CYLC_SUITE_DEF_PATH) or in the suite bin directory (via $PATH). If this item is not defined, the local suite definition directory path will be assumed, with the suite owner’s home directory, if present, replaced by '$HOME' for interpretation on the remote host.
[runtime] → [[__NAME__]] → [[[event hooks]]] See Section A.2.7 (Suite Event Hooks) for a general description of cylc event handling. This section is specific to task events. The command cylc [hook] email-task is a ready-made task event handler.
Custom task event handlers can be located in the suite bin directory, in which case you will not need to modify your $PATH to ensure they are found. They are called by cylc with the following arguments:
EVENT is the event name (see below), SUITE is the suite name, TASK is the task ID, and MESSAGE, if provided, describes what has happened. Note that spaces in event names will be replaced by underscores in the handler argument list to make parsing easier in the script, e.g. “submission failed” becomes “submission_failed”.
[runtime] → [[__NAME__]] → [[[event hooks]]] → EVENT handler The handler to call when the task event EVENT occurs. Repeat this section for each of the following task events that you wish to handle:
MESSAGE, if provided, describes what has happened, and TASKID identifies the task (NAME.CYCLE for cycling tasks).
To handle timeouts you must also specify a timeout value, below.
[runtime] → [[__NAME__]] → [[[event hooks]]] → submission timeout If a task has not started the specified number of minutes after it was submitted, the event handler will be called by cylc with submission_timeout as the EVENT argument:
[runtime] → [[__NAME__]] → [[[event hooks]]] → execution timeout If a task has not finished the specified number of minutes after it started running, the event handler will be called by cylc with execution_timeout as the EVENT argument:
[runtime] → [[__NAME__]] → [[[event hooks]]] → reset timer If you set an execution timeout the timer can be reset to zero every time a message is received from the running task (which indicates the task is still alive). Otherwise, the task will timeout if it does not finish in the alotted time regardless of incoming messages.
[runtime] → [[__NAME__]] → [[[environment]]] The user defined task execution environment. Variables defined here can refer to cylc suite and task identity variables, which are exported earlier in the task job script, and variable assignment expressions can use cylc utility commands because access to cylc is also configured earlier in the script. See also Task Execution Environment, Section 8.4.7.
[runtime] → [[__NAME__]] → [[[environment]]] → __VARIABLE__ Replace __VARIABLE__ with any number of environment variable assignment expressions. Order of definition is preserved so values can refer to previously defined variables. Values are passed through to the task job script without evaluation or manipulation by cylc, so any variable assignment expression that is legal in the job submission shell can be used. White space around the ‘=’ is allowed (as far as cylc’s suite.rc parser is concerned these are just normal configuration items).
[runtime] → [[__NAME__]] → [[[directives]]] Batch queue scheduler directives. Whether or not these are used depends on the job submission method. For the built-in loadleveler, pbs, and sge methods directives are written to the top of the task job script in the correct format for the method. Specifying directives individually like this allows use of default directives that can be individually overridden at lower levels of the runtime namespace hierarchy.
[runtime] → [[__NAME__]] → [[[directives]]] → __DIRECTIVE__ Replace __DIRECTIVE__ with each directive assignment, e.g. class = parallel
Example directives for the built-in job submission methods are shown in Section 10.2.
[runtime] → [[__NAME__]] → [[[outputs]]] This section is only required if other tasks need to trigger off specific internal outputs of this task (as opposed to triggering off it finishing). The task implementation must report the specified output messages by calling cylc task message when the corresponding real outputs have been completed.
[runtime] → [[__NAME__]] → [[[outputs]]] → __OUTPUT__ Replace __OUTPUT__ with any number of labelled output messages.
where the item name must match the output label associated with this task in the suite dependency graph, e.g.:
Configuration of suite graphing and, where explicitly stated, the graph-based suite control GUI.
The first cycle time to use when plotting the suite dependency graph.
The last cycle time to use when plotting the suite dependency graph. Typically this should be just far enough ahead of the initial cycle to show the full suite.
A list of family (namespace) names to be shown in the collapsed state (i.e. the family members will be replaced by a single family node) when the suite is plotted in the graph viewer or the gcylc graph view. This item determines how family groups are shown initially in the suite control GUI; subsequently you can use the interactive controls to group and ungroup nodes at will. For the same reason (presence of interactive grouping controls) this item is ignored if the suite is reparsed during graph viewing (other changes to graph styling will be picked up and applied if the graph viewer detects that the suite.rc file has changed).
Graph edges (dependency arrows) can be plotted in the same color as the upstream node (task or family) to make paths through a complex graph easier to follow.
Graph node labels can be printed in the same color as the node outline.
Set the default attributes (color and style etc.) of graph nodes (tasks and families). Attribute pairs must be quoted to hide the internal = character.
Set the default attributes (color and style etc.) of graph edges (dependency arrows). Attribute pairs must be quoted to hide the internal = character.
If True, the graph-based suite control GUI will write out a dot-language graph file on every change; these can be post-processed into a movie showing how the suite evolves. The frames will be written to the run time graph directory (see below).
Define named groups of graph nodes (tasks and families) which can styled en masse, by name, in [visualization] → [[node attributes]]. Node groups are automatically defined for all task families, including root, so you can style family and member nodes at once by family name.
[visualization] → [[node groups]] → __GROUP__ Replace __GROUP__ with each named group of tasks or families.
Here you can assign graph node attributes to specific nodes, or to all members of named groups defined in [visualization] → [[node groups]] (task families are automatically node groups). Group styling can be overridden for individual nodes or subgroups.
[visualization] → [[node attributes]] → __NAME__ Replace __NAME__ with each node or node group for style attribute assignment.
Cylc can generate graphs of dependencies resolved at run time, i.e. what actually triggers off what as the suite runs. This feature is retained mainly for development and debugging purposes. You can use simulation mode or dummy mode to generate runtime graphs very quickly.
[visualization] → [[runtime graph]] → enable Runtime graphing is disabled by default.
[visualization] → [[runtime graph]] → cutoff New nodes will be added to the runtime graph as the corresponding tasks trigger, until their cycle time exceeds the initial cycle time by more than this cutoff, in hours.
[visualization] → [[runtime graph]] → directory Where to put the runtime graph file, runtime-graph.dot.
See Section 8.7.
Cylc provides, via $CYLC_DIR/conf/suiterc/⋆.spec, sensible default values for many configuration items so that most users will not need to explicitly configure log directories and so on. The defaults are sufficient, in fact, to allow test suites defined by dependency graph alone (command scripting, for example, defaults to printing a simple message, sleeping for a few seconds, and then exiting).
The cylc get-config command parses a suite definition and retrieves configuration values for individual items, sections, or entire suites.
Cylc now has a site config file for settings that apply to all suites: $CYLC_DIR/conf/site/site.rc. Prior to cylc-5.0 some of these settings (e.g. log directory locations) had to be put in suite definitions, and some (e.g. preferred editors) were specified in the user’s environment. Many of the site settings can be overridden by users: $HOME/.cylc/user.rc
See also cylc get-global-config --help, which can be used to write an initial site or user config file with internal documentation and all items commented out.
As a temporary measure the self-documenting configspec for site/user config files is included verbatim here.
Each cylc user can optionally run his/her own lockserver to prevent accidental invocation of multiple instances of the same suite or task at the same time. The suite and task locks brokered by the lockserver are analogous to traditional lock files, but they work across a network, even for distributed suites containing tasks that start executing on one host and finish on another.
Accidental invocation of multiple instances of the same suite or task at the same time can have serious consequences, so use of the lockserver should be considered for important operational suites, but it may be considered an unnecessary complication for general less critical usage, so it is currently disabled by default.
To enable the lockserver:
The suite will now abort at start-up if it cannot connect to the lockserver. To start your lockserver daemon,
To check that it is running,
For detailed usage information,
There is a command line client interface,
for interrogating the lockserver and managing locks manually (e.g. releasing locks if a suite was killed before it could clean up after itself).
To watch suite locks being acquired and released as a suite runs,
The graph view in the gcylc GUI has the advantage that it shows the structure of a suite very clearly as it evolves. It works remarkably well even for very large suites (up to several hundred tasks or more) but because the graphviz engine does a new global layout every time the graph changes the layout is often not very stable. This may not be a solvable problem even in principle as it seems likely that making continual incremental changes to an existing graph without redoing the global layout would inevitably result in a horrible mess.
The following features of the graph view, however, help mitigate the the jumping layout problem:
Early versions of cylc were focused on developing and testing the new scheduling algorithm, and the suite design interface at the time was essentially the quickest route to that end. A suite was a collection of “task definition files” that encoded the prerequisites and outputs of each task in a direct reflection of cylc’s internal task proxies. This way of defining suites exposed cylc’s self-organising nature to the user, and it did have some nice properties. For instance a group of tasks could be transferred directly from one suite to another by simply copying the taskdef files over (and checking that prerequisite and output messages were consistent with the new suite). However, ensuring consistency of prerequisites and outputs across a large suite could be tedious; a few edge cases associated with suite start-up and forecast model restart dependencies were, arguably, difficult to understand; and the global structure of a suite was not readily apparent until run time (although to counter this cylc 2.x could generate run-time resolved dependency graphs very quickly in simulation mode).
Version 3.0 implemented an entirely new suite design interface in which one defines the suite dependency graph, execution environment, and command scripting for each task, in a single structured, validated, configuration file - the suite.rc file. This makes suite structure apparent at a glance, and much important detail is now implicitly implied by the graph.
Version 4.0 has the following major improvements over cylc-3.x, along with many refinements:
Version 5.0 contains some major internal changes to enhance performance for large suites, such as multi-threading for continuous request handling and task job submission. We also aim to provide backward compatibility for suite definitions from version 5.0 onward, wherever possible.
Pyro (Python Remote Objects) is a widely used open source objected oriented Remote Procedure Call technology developed by Irmen de Jong.
Earlier versions of cylc used the Pyro Nameserver to marshal communication between client programs (tasks, commands, viewers, etc.) and their target suites. This worked well, but in principle it provided a route for one suite or user on the subnet to bring down all running suites by killing the nameserver. Consequently cylc now uses Pyro simply as a lightweight object oriented wrapper for direct network socket communication between client programs and their target suites - all suites are thus entirely isolated from one another.
Copyright Ⓒ 2007 Free Software Foundation, Inc. http://fsf.org/
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for software and other kinds of works.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program–to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.
Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it.
For the developers’ and authors’ protection, the GPL clearly explains that there is no warranty for this free software. For both users’ and authors’ sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions.
Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users’ freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and modification follow.
Terms and Conditions
“This License” refers to version 3 of the GNU General Public License.
“Copyright” also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
“The Program” refers to any copyrightable work licensed under this License. Each licensee is addressed as “you”. “Licensees” and “recipients” may be individuals or organizations.
To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work.
A “covered work” means either the unmodified Program or a work based on the Program.
To “propagate” a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To “convey” a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays “Appropriate Legal Notices” to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
The “source code” for a work means the preferred form of the work for making modifications to it. “Object code” means any non-source form of a work.
A “Standard Interface” means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The “System Libraries” of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A “Major Component”, in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The “Corresponding Source” for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work’s System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work’s users, your or third parties’ legal rights to forbid circumvention of technological measures.
You may convey verbatim copies of the Program’s source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an “aggregate” if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation’s users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A “User Product” is either (1) a “consumer product”, which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, “normally used” refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
“Installation Information” for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
“Additional permissions” are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
All other non-permissive additional terms are considered “further restrictions” within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An “entity transaction” is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party’s predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
A “contributor” is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor’s “contributor version”.
A contributor’s “essential patent claims” are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, “control” includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor’s essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a “patent license” is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To “grant” such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. “Knowingly relying” means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient’s use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is “discriminatory” if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such.
The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License “or any later version” applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy’s public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
End of Terms and Conditions
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the “copyright” line and a pointer to where the full notice is found.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode:
The hypothetical commands show w and show c should show the appropriate parts of the General Public License. Of course, your program’s commands might be different; for a GUI interface, you would use an “about box”.
You should also get your employer (if you work as a programmer) or school, if any, to sign a “copyright disclaimer” for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see http://www.gnu.org/licenses/.
The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read http://www.gnu.org/philosophy/why-not-lgpl.html.
1Future plans for EcoConnect include additional deterministic regional weather forecasts and a statistical ensemble.
2In fact this dependency negotiation goes through a broker object (rather than every task literally checking every other task) which scales as n (rather than n2) where n is the number of task proxies in the pool.
3Sections are closed by the next section heading, so items within a section must be defined before any subsequent subsection headings.
4The exceptions were designed to allow tasks to override environment variables defined in include-files that could be included in multiple tasks, to assist in factoring out common task configuration. However, namespace inheritance now provides a better way to do this in most cases.
5In NWP forecast analysis suites parts of the observation processing and data assimilation subsystem will typically also depend on model background fields generated by the previous forecast.
6An OR operator on the right doesn’t make much sense: if “B or C” triggers off A, what exactly should cylc do when A finishes?
7A warm cycling model that only writes out one set of restart files, for the very next cycle, does not need to be declared sequential because this early triggering problem cannot arise.
8Note that $CYLC_SUITE_ENVIRONMENT is a string containing embedded newline characters and it has to be handled accordingly. In the bash shell, for instance, it should be echoed in quotes to avoid concatenation to a single line.
9The cylc submit command runs a single task exactly as its suite would, in terms of both job submission method and execution environment.
10If you copy a suite using cylc commands or db viewer the entire suite definition directory will be copied.
11Spawning any earlier than this brings no advantage in terms of functional parallelism and would cause uncontrolled proliferation of waiting tasks.
12This is because you don’t want Model[T] waiting around to trigger off Model[T-12] if Model[T-6] has not finished yet. If Model is forced to be sequential this can’t happen because Model[T] won’t exist in the suite until Model[T-6] has finished. But if Model[T-6] fails, it can be spawned-and-removed from the suite so that Model[T] can then trigger off Model[T-12], which is the correct behaviour.
]
